It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification problem. However, you don't always have lots of data to train your machine learning algorithm.
Which classical machine learning methods work well with little data? I'm thinking of things like KNN, linear regression, support vector machines, random forests, etc.
Is there a paper that systematically investigates machine learning methods when data are scarse? If not, are there some rules of thumb one can follow?