0
$\begingroup$

It is well-known that deep neural networks require lots of data to perform reliably and well. A commonly-cited statistic is that you need at least 10,000 examples per class for a classification problem. However, you don't always have lots of data to train your machine learning algorithm.

Which classical machine learning methods work well with little data? I'm thinking of things like KNN, linear regression, support vector machines, random forests, etc.

Is there a paper that systematically investigates machine learning methods when data are scarse? If not, are there some rules of thumb one can follow?

Improve this question
$\endgroup$
2
  • $\begingroup$ Currently to train state-of-the-art robots for bicycling needs millions of trials from a blank slate on average compared with much less trials for average young aged human. You may contemplate the root cause here. There seems no other way around this issue until we have true AGI to have agent continuously evolved to learn new objectives/rewards based upon previous learned policies in the RL formulation. $\endgroup$
    – cinch
    Commented Dec 16, 2022 at 21:32
  • $\begingroup$ You are asking for sample efficiency of learning algorithms $\endgroup$
    – Ggjj11
    Commented May 4 at 21:12

1 Answer 1

Reset to default
1
$\begingroup$

Generally speaking, the number of training examples needed to train a neural network for your classification task cannot be determined a priori. Generalizations such as the claim that 10K training examples needed are obtuse. This question is complicated, but here are some considerations:

  1. Technically, you can generate a model with very few training examples (equal to the number of classes), but model performance will be poor. Thus, it is not sufficient to ask how many training examples are needed. It would be more apropos to ask how many are needed to achieve a target performance level.
  2. The number of examples depends on the complexity of your task, the expressivity of your model, and the quality of your data.
  • How many classes do you have (binary OR 100 classes)?
  • How different are the classes from each other (snake vs plane OR python vs boa)?
  • What types of features do you have (imaging, time series, tabular data)?
  • How much label noise is in the data?
  • How much feature noise is in the data?
  • How predictive are the features of the classes (relates to #1)?
  • What is the capacity (model complexity) of your model to learn the pattern?
  1. In a more practical sense, I search the literature for publications that worked on similar tasks as my task and see what models they used, how many examples they used, and what type of performance they achieved.
  2. Consider expanding your training data with Data Augmentation or GANs or public data sets or working with collaborators.
  3. Consider transfer learning if you have scarce data.
  4. Besides neural networks, there are again no a priori methods for determining which traditional ML model is going to perform better. You have to empirically test them out.
  5. Consider N-shot learning.
  6. Depending on whether you have very little data over all or whether you have very little labeled data, considered semi-supervised learning.
Improve this answer
$\endgroup$
1
  • $\begingroup$ I agree with most! Only a minor point: data augmentation can be harmful. Only in the setting of multi objective optimization (like optimizing e.g. recall of multiple classes etc.) data augmentation is a very indirect way of changing the tradeoffs $\endgroup$
    – Ggjj11
    Commented May 4 at 21:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .