Research paths/areas for improving the performance of CNNs when faced with limited data

Question

I've been reading through the research literature for image processing, computer vision, and convolutional neural networks. For image classification and object recognition, I know that convolutional neural networks deliver state-of-the-art performance when large amounts of data are available. Furthermore, I know that Hinton et al. created "capsule networks" to try and overcome some of the fundamental limitations of CNN architecture (such as them not being rotationally invariant). However, my understanding is that capsule networks have been a failure (so far), and most people expect them to go nowhere. And CNNs have progressively been improved in various ways (Bayesian optimisation for hyper parameter tuning, new convolution kernels, etc.). It seems to me that, at the moment, and for the foreseeable future, CNNs are the best architecture available for image-related stuff.

But, as I said, CNNs, like other Deep Learning architectures, require large amounts of data. So my question is as follows:

What are the research areas/topics for improving CNNs in the sense of making them work more effectively (that is, have greater performance) with less data (working with small datasets)?

I know that there is various research looking at approaches to increasing data (such as data augmentation, generative networks, etc.), but I am primarily interested in fundamental modifications to CNNs themselves, rather than purely focusing on changes to the data itself.

And to expand upon my question, using my above definition of "performance", I am interested in these two categories:

1. "Computational methods" for increasing CNN performance. This would be the non-mathematical stuff that I've read about, such as just increasing the number of layers and making the CNN deeper/wider (and I think another one had to do with just making the size of the convolution kernel smaller, so that it looks at smaller pieces of the image at any one time, or something like that?).

2. "Mathematical methods" for increasing CNN performance. This would be the cutting-edge mathematical/statistical stuff that I've read about: things like algorithms (such as Bayesian optimization); I've come across a lot of geometric stuff; and I guess the cutting-edge convolution kernels created by the image processing people would also fall under this category.

Obviously, this "list" is not exhaustive, and it's probably incorrect; I'm a novice to this research, so I'm trying to find my way around.

I am interested in studying both of the above categories, but I will primarily be working from the mathematical/statistical side. And I want to work on research that is still practical and can be put to use in industry for improved performance (even if it might still be "advanced"/complex for most people in industry) – not the the highly theoretical stuff related.

Related (but unanswered): Are there any good research papers on image identification with limited data?

Answer 1

Some research areas that come to mind which can be useful when faced with a limited amount of data:

• Regularization: Comprises different methods to prevent the network from overfitting, to make it perform better on the validation data but not necessarily on the training data. In general, the less training data you have, the stronger you want to regularize. Common types include:

  • Injecting noise in the network, e.g., dropout.

  • Adding regularization terms to the training loss, e.g., L1 and L2 regularization of the weights, but also confident output distributions can be penalized.

  • Reducing the number of parameters in the network to make it unable to fit the training data completely and thus unable to overfit badly. Interestingly, increasing the number of parameters for large models can also improve the validation performance.

  • Early stopping of training. For example, if one part of the training set is set aside and not used to update the weights, training can be stopped when the observed loss on this part of the training set is observed to start to increase.

• Generating new training data:

  • Data augmentation: Ways to augment existing training examples without removing the semantics, e.g., slight rotations, crops, translations (shifts) of images.

  • Data interpolation, e.g., manifold mixup.

  • Using synthetic data, e.g., frames from video games or other CGI.

• Transfer learning: When you take a neural network that has already been trained on another, much larger dataset of the same modality (images, sounds, etc.) as your dataset and fine-tune it on your data.

• Multitask learning: Instead of training your network to perform one task, you give it multiple output heads and train it to perform many tasks at once, given that you have that the labels for the additional tasks. While it may seem that this is a more difficult for the network, the extra tasks have a regularizing effect.

• Semi-supervised learning: If you have much unlabeled data that labeled data, you can combine supervised learning with unsupervised learning. Much like with multitask learning, the extra task introduced by the unsupervised learning also has a regularizing effect.

Other interesting methods can be found in systems that perform one-shot learning, which inherently implies very little training data. These systems often uses slightly modified network architectures. For example, facial recognition systems can learn to recognize a face from only a single photo, and usually use a triplet loss (or similar) of a vector encoding of the face, instead of cross-entropy loss of the output of a softmax layer normally used for image classification.

Zero-shot learning also exists (e.g., zero-shot machine translation), but this is a completely different type of problem setup and requires multiple data modalities.