0
$\begingroup$

When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?

I have an DNN training from an infinite stream of samples, that most likely won't repeat. So there is no real notion of "epoch".

Now I wonder if the math behind ADAM or other popular optimizers expect the repetition of data over the epochs?

If so, should I collect a limited amount of those samples and use them for training data and validation data, or would it be better to use all data available (even if the training data never repeats then)?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have infinitely many images of dogs and cats, there are features that discriminate the two animals that are mostly consistent like the shape of the nose. Having infinitely many samples in these cases is generally desirable, because it can benefit the ability of the model to generalize.

In contrast, there are cases where this is not true that depend on the data: If your data contains (concept-) drift, meaning that your data distribution changes over time, the model you train might not be able to learn a consistently performing function and therefore chases a moving objective. For images, this can be the case if the labels would depend on lighting conditions that constantly change (concept drift), or if the objects in the images that you want to classify continuously change shape (drift).

Improve this answer
$\endgroup$
2
  • $\begingroup$ I assume my data has no 'drift', the first training dataset will serve the purpose as well as any later one. I am just wonder if ADAM and other methods somehow assume the same dataset for consecutive batches or something to be able to tune their internal parameters. $\endgroup$
    – dronus
    Commented Feb 3, 2023 at 13:06
  • $\begingroup$ No they don't do that, they are purely local functions, i.e. they use the current batch and don't keep any sort of 'record' of the whole dataset. With the exception of the momentum term, but that one doesn't require your dataset to repeat at some point. $\endgroup$
    – Chillston
    Commented Feb 3, 2023 at 13:34
0
$\begingroup$

Subdividing datasets into batches is only done for computational reasons (RAM, computation time etc). Optimizers do not care about repeated data. They get a batch of data to optimize on and will calculate the gradient for that batch and update the model accordingly. Nothing in this process requires the batches to reappear at all.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I agree in general but dividing data into batches also introduces noise (since the batch does represent the probability distribution of the whole dataset only up to a certain extent, the difference can be interpreted as noise) and might therefore help to overcome local minima $\endgroup$
    – pythonic833
    Commented Feb 14, 2023 at 13:53
  • 1
    $\begingroup$ I never pretended that having smaller or bigger batches does not matter in an optimization process. They definitely do. However, whether batches reappear exactly as is should not matter for optimization, which was what the author asked. $\endgroup$
    – Robin van Hoorn
    Commented Feb 14, 2023 at 14:53
  • 1
    $\begingroup$ I just wanted to add this since the formulation in your post says "is only done for computational reasons" which IMHO neglects the theoretical aspect of introducing noise. $\endgroup$
    – pythonic833
    Commented Feb 14, 2023 at 15:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .