I've been reading through the research literature for image processing, computer vision, and convolutional neural networks. For image classification and object recognition, I know that convolutional neural networks deliver state-of-the-art performance when large amounts of data are available. Furthermore, I know that Hinton et al. created "capsule networks" to try and overcome some of the fundamental limitations of CNN architecture (such as them not being rotationally invariant). However, my understanding is that capsule networks have been a failure (so far), and most people expect them to go nowhere. And CNNs have progressively been improved in various ways (Bayesian optimisation for hyper parameter tuning, new convolution kernels, etc.). It seems to me that, at the moment, and for the foreseeable future, CNNs are the best architecture available for image-related stuff.
But, as I said, CNNs, like other Deep Learning architectures, require large amounts of data. So my question is as follows:
What are the research areas/topics for improving CNNs in the sense of making them work more effectively (that is, have greater performance) with less data (working with small datasets)?
I know that there is various research looking at approaches to increasing data (such as data augmentation, generative networks, etc.), but I am primarily interested in fundamental modifications to CNNs themselves, rather than purely focusing on changes to the data itself.
And to expand upon my question, using my above definition of "performance", I am interested in these two categories:
"Computational methods" for increasing CNN performance. This would be the non-mathematical stuff that I've read about, such as just increasing the number of layers and making the CNN deeper/wider (and I think another one had to do with just making the size of the convolution kernel smaller, so that it looks at smaller pieces of the image at any one time, or something like that?).
"Mathematical methods" for increasing CNN performance. This would be the cutting-edge mathematical/statistical stuff that I've read about: things like algorithms (such as Bayesian optimization); I've come across a lot of geometric stuff; and I guess the cutting-edge convolution kernels created by the image processing people would also fall under this category.
Obviously, this "list" is not exhaustive, and it's probably incorrect; I'm a novice to this research, so I'm trying to find my way around.
I am interested in studying both of the above categories, but I will primarily be working from the mathematical/statistical side. And I want to work on research that is still practical and can be put to use in industry for improved performance (even if it might still be "advanced"/complex for most people in industry) – not the the highly theoretical stuff related.
Related (but unanswered): Are there any good research papers on image identification with limited data?