4
$\begingroup$

Open AI spin up says

... the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning.

  1. Why would recent data overfit? Is it because they are more correlated to each other so I am training on a bunch of similar data?

  2. Why is it slow to use much memory (besides the simple reason that I am storing more data). Below are specific doubts I have.

    1. I thought convergence is faster when fitting on uncorrelated data. For example, estimates on mean. Recent data would be more correlated. So why would training on longer data slow down training?

    2. Previous samples doesn't seem to have the notion of becoming outdated. They are still valid samples for evaluating the current loss. So why would they slow down training?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Please, ask only one question per post. If you have multiple questions, ask each in its separate post with the appropriate context. I see many ? here, so it seems you're asking many questions. If you have one main question and the other are just sub-questions that help people answer your main question, then you should probably clarify that. I suggest that you also fix the title, which ends with "and", so it seems that you missed something there. $\endgroup$
    – nbro
    Commented Jun 14, 2022 at 10:34
  • $\begingroup$ Your question(s) is/are related to this. $\endgroup$
    – nbro
    Commented Jun 15, 2022 at 9:14
  • $\begingroup$ Thanks for your advice on the post, I edited the question and took out some part for another post. I did read the question you referenced, but although it is on the same topic, the answers there didn't solved the question for me, so I try to list out specific points I am confused about here. $\endgroup$
    – Sara
    Commented Jun 16, 2022 at 1:30

1 Answer 1

Reset to default
3
$\begingroup$

I read the same thing recently, and my interpretation was this:

If you only use the very-most recent data, you will overfit to that and things will break

We'd like to train the network to predict accurately for all states and actions. If we only show it recent data, we're only showing it a subset of the possible states and actions, so it might overfit those and forget how to predict in other parts of the environment.

Similar to training a model to classify images of cats, dogs, and elephants, but only showing it cats and dogs for a while. It can forget what elephants look like.

if you use too much experience, you may slow down your learning.

Specifically, I believe, when encountering novel data. With more memory, the novel data will be chosen for updates less often. The network will learn more slowly, because it spends much of its time reviewing old data that it has already learned.

The optimal balance feeds the network as much new data as possible while feeding it enough old data so it doesn't forget.

Improve this answer
$\endgroup$
2
  • $\begingroup$ this is helpful in general. But could you respond to the specific points I raised as well? $\endgroup$
    – Sara
    Commented Jun 16, 2022 at 1:42
  • $\begingroup$ I believe the points you made are accurate regarding convergence of a NN to fixed training data, but this training data is not fixed. In later learning steps the NN has already converged on the majority of states (losses are near zero), and the NN is only learning from a few novel states. Keeping old data slows down training because it reduces how often the only states the network can learn from (the novel states) are included in a minibatch. (At least that's my understanding.) $\endgroup$
    – Lee Reeves
    Commented Jun 16, 2022 at 2:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .