5
$\begingroup$

I am working on a project where I have to classify around 1000 unique objects. I'm trying to plan how much training data I will need to collect. I was planning on using ResNet-50. Is there anyway I can estimate the amount of photos I should plan to collect ahead of time (assuming I will collect an equal amount of photos of each class)?

$\endgroup$

1 Answer 1

6
$\begingroup$

What you want to calculate/estimate is known as the sample complexity in computational learning theory. If you knew the VC dimension of the neural network, you may be able to estimate the sample complexity, but your estimate (bound) may not be very tight anyway (maybe because the estimate of the VC dimension is also not tight). Here is more info about the VC dimension and the VC dimension of a few neural networks.

If we were able to easily compute (an accurate estimate of) the sample complexity in all cases, this would be extremely useful, as we could exclude (or not) that the size of the training dataset is the problem of under-fitting/over-fitting.

Nowadays, I think that most people do not usually estimate the sample complexity (I could be wrong, i.e. there may be simple ways to do this approximately, apart from saying that you probably need more training data points than the number of parameters, which is what many practitioners will tell you, maybe misleadingly), but, instead, collect as much data as possible, then train the neural network with that data; if it performs poorly on the test dataset, then you may try to collect more data (given that it's an indication that the neural network did not learn the target function); if you see that the test performance initially increases but then decreases while the training performance is always increasing, it may be an indication of over-fitting, so you may want to use a regularisation technique (like dropout), which you should probably use anyway, especially when you have a small dataset.

I also want to point out that, with the discovery of the phenomena double descent and grokking, these traditional guidelines or practices (e.g. early stopping) may sometimes be misleading (I don't know when or why), i.e. just keep in mind that sometimes they may not be applicable.

$\endgroup$
1
  • 1
    $\begingroup$ Worth adding a standard way to assess whether more data would help = plotting a learning curve where the x axis is amount of data used to train, and y axis is performance of the model on a (fixed) validation set. If it plateaus, then more data may not help (with the specific model being tested), if it is still trending upwards, more data probably will help. $\endgroup$ Nov 17, 2021 at 8:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .