2
$\begingroup$

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

What does this mean? Is it related to the tuning of the parameter of the batch size in the algorithm?

$\endgroup$
  • $\begingroup$ Hi Tracy! Can you please just ask one question per post? I would ask the question about the target network in another post. $\endgroup$ – nbro Apr 4 at 14:54
  • $\begingroup$ By the way, questions about target networks and experience replay has already been asked on this website in the past. See respectively ai.stackexchange.com/q/6982/2444 and ai.stackexchange.com/q/6579/2444. $\endgroup$ – nbro Apr 4 at 14:58
  • $\begingroup$ @nbro,thanks for the information. I will remove the target network part, but keep the questions about replay buffer. $\endgroup$ – Tracy Yang Apr 4 at 15:12
  • $\begingroup$ I just added an answer to the other question regarding the replay buffer. Have a look at it: ai.stackexchange.com/a/11644/2444. If it clarifies your doubts, I think you should delete this question, because it is pretty much a duplicate of the other. $\endgroup$ – nbro Apr 4 at 15:46
  • $\begingroup$ @nbro, thanks for the answer. Then how about the second question related to the quote? Does that mean we need to tune the size of mini-batch, i.e., how many tuples we need to sample from the replay buffer? $\endgroup$ – Tracy Yang Apr 4 at 16:13
3
$\begingroup$

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.

The larger the experience replay, the less likely you will sample correlated elements, hence the more stable the training of the NN will be. However, a large experience replay also requires a lot of memory and it might slow training. So, there is a trade-off between training stability (of the NN) and memory requirements.

The authors of the linked article state (right after the sentence above)

If you only use the very-most recent data, you will overfit to that and things will break; if you use too much experience, you may slow down your learning. This may take some tuning to get right.

$\endgroup$
  • $\begingroup$ Large replay buffer doesn't necessarily slow down training. It may actually accelerate it by preventing overfitting on near-policy samples. The only real drawback is memory usually. $\endgroup$ – mirror2image Jul 10 at 17:40
  • $\begingroup$ @mirror2image You might be right, so I updated the answer. $\endgroup$ – nbro Jul 10 at 17:46
1
$\begingroup$

To add to the answer by @nbro

Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.

Such a buffer is simply a way to "see" what was up to 100k iterations ago. After the first 100k iterations you fill the buffer and begin "moving" it, much like a sliding window, by inserting new memories instead of the oldest.


The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on "how much you believe your network architecture is susceptible to catastrophic forgetting".

A tiny buffer might force your network to only care about what it saw recently.

But an excessively large buffer might take a long time to "become refreshed" with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.

The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.

$\endgroup$
  • 1
    $\begingroup$ Good point. Just to add to why this is important: this is important because old experience was generated by an old policy (i.e. your data distribution consists of an older distribution over state-action pairs / trajectories), and because any stored value estimates may be old. Which one of those two points matters depends on whether we're talking about value-based or policy gradient methods, and whether we're talking about on-policy or off-policy learning... but usually at least one of those two is relevant $\endgroup$ – Dennis Soemers Jul 10 at 10:46
  • 2
    $\begingroup$ @DennisSoemers: Generally you should not store value estimates in experience replay, but re-calculate values when using the trajectory information (s,a,r,s'). No doubt there are implementations which store value estimates for some valid reason, but "classic" DQN does not $\endgroup$ – Neil Slater Jul 10 at 11:39
  • $\begingroup$ @NeilSlater Yeah you're right, I guess you shouldn't in most (all?) of the classic RL algorithms. I've been working a lot with AlphaZero-style self-playing training the past year, and there we do store value estimates as computed by MCTS... so I guess that's why I've got it stuck in my mind :) $\endgroup$ – Dennis Soemers Jul 10 at 11:51
  • $\begingroup$ No wait that's wrong too, in standard AlphaZero-style training we store visit counts, but in one of my recent papers I changed that to storing MCTS' value estimates... $\endgroup$ – Dennis Soemers Jul 10 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.