15
$\begingroup$

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

What does this mean? Is it related to the tuning of the parameter of the batch size in the algorithm?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
11
$\begingroup$

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" They explicitly test the size of the experience replay, the replay-ratio of each experience and other parameters.

Also, to add to the answer by @nbro

Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.

Such a buffer is simply a way to "see" what was up to 100k iterations ago. After the first 100k iterations you fill the buffer and begin "moving" it, much like a sliding window, by inserting new memories instead of the oldest.

The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on "how much you believe your network architecture is susceptible to catastrophic forgetting".

A tiny buffer might force your network to only care about what it saw recently.

But an excessively large buffer might take a long time to "become refreshed" with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.

The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Good point. Just to add to why this is important: this is important because old experience was generated by an old policy (i.e. your data distribution consists of an older distribution over state-action pairs / trajectories), and because any stored value estimates may be old. Which one of those two points matters depends on whether we're talking about value-based or policy gradient methods, and whether we're talking about on-policy or off-policy learning... but usually at least one of those two is relevant $\endgroup$
    – Dennis Soemers ♦
    Commented Jul 10, 2019 at 10:46
  • 2
    $\begingroup$ @DennisSoemers: Generally you should not store value estimates in experience replay, but re-calculate values when using the trajectory information (s,a,r,s'). No doubt there are implementations which store value estimates for some valid reason, but "classic" DQN does not $\endgroup$
    – Neil Slater
    Commented Jul 10, 2019 at 11:39
  • $\begingroup$ @NeilSlater Yeah you're right, I guess you shouldn't in most (all?) of the classic RL algorithms. I've been working a lot with AlphaZero-style self-playing training the past year, and there we do store value estimates as computed by MCTS... so I guess that's why I've got it stuck in my mind :) $\endgroup$
    – Dennis Soemers ♦
    Commented Jul 10, 2019 at 11:51
  • $\begingroup$ No wait that's wrong too, in standard AlphaZero-style training we store visit counts, but in one of my recent papers I changed that to storing MCTS' value estimates... $\endgroup$
    – Dennis Soemers ♦
    Commented Jul 10, 2019 at 11:57
10
$\begingroup$

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.

The authors of the linked article state (right after the sentence above)

If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.

Improve this answer
$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .