3
$\begingroup$

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we discussed in Chapter 6, Deep Q-Network, when we talked about the correlation between samples, which breaks the independent and identically distributed (i.i.d) assumption, which is critical for Stochastic Gradient Descent (SDG) optimization. The negative consequence of such correlation is very high variance in gradients, which means that our training batch contains very similar examples, all of them pushing our network in the same direction. However, this may be totally the wrong direction in the global sense, as all those examples could be from one single lucky or unlucky episode. With our Deep Q-Network (DQN), we solved the issue by storing a large amount of previous states in the replay buffer and sampling our training batch from this buffer. If the buffer is large enough, the random sample from it is much better representation of the states distribution at large. Unfortunately, this solution won't work for PG methods, at most of them are on-policy, which means that we have to train on samples generated by our current policy, so, remembering old transitions is not possible anymore.

The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 284.

How being on-policy preventing us from using the replay buffer with the PG? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?

$\endgroup$
  • $\begingroup$ I answered similar question here you can take a look $\endgroup$ – Brale May 15 at 16:53
  • $\begingroup$ I am not sure of your answer. Are you up to make a full answer using a mathematical explanation? $\endgroup$ – jgauth May 15 at 17:10
  • $\begingroup$ What part do you not understand ? $\endgroup$ – Brale May 15 at 17:22
3
$\begingroup$

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|S_t = s] &= E_{\pi_b}[R_{t+1} + G_{t+1}]\\ &= \sum_a \pi_b(a|s) \sum_{s', r} p(s', r|s, a) [r + E_{\pi_b}[G_{t+1}|S_{t+1} = s']]\\ \end{align} You can see if you write out this recursively that this expectation depends on $\pi_b(a|s), \pi_b(a'|s'), \ldots$

If you collect trajectory with policy $\pi_a$ you would get expected return that depends on $\pi_a(a|s), \pi_a(a'|s'), \ldots$ Since these are two different policies then $\pi_b(a|s) \neq \pi_a(a|s)$ for some $(s, a)$. That would mean that returns have different expected values and are sampled through different distributions. You cannot then use some return $G$ sampled by following policy $\pi_b$ to update policy $\pi_a$ because it's not sampled according to the proper distribution, if we did, we would be updating policy $\pi_a$ with biased gradient update that does not reflect how policy $\pi_a$ performed.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Awesome answer @Brale_! Can you do me a little favor unrelated to this question? $\endgroup$ – jgauth May 15 at 19:45
  • $\begingroup$ You can ask but I cannot really promise you anything though. $\endgroup$ – Brale May 15 at 19:51
  • 1
    $\begingroup$ Or in more plain, non-mathematical terms; if you're traing on old examples stored in some replay buffer, which were created by another / an older policy, you're by definition doing off-policy learning! $\endgroup$ – Dennis Soemers May 15 at 19:57
  • $\begingroup$ Ok. I am working with the Max Lapan's book, i.e. Deep Reinforcement Learning Hands-on on the subject of A2C. I tried to build my own implementation of the Atari Pong game using that algo and based on Max code, but my code is not converging at all. I'm stuggling on that converging issue since a week now. I think there's a very subtle error in the code, but I can't say what it is. Here is the question : ai.stackexchange.com/questions/21187/…. Can you help me with that? $\endgroup$ – jgauth May 15 at 19:58
  • 1
    $\begingroup$ for example in numpy you can call numpy.random.seed and your random numbers will always follow same pattern every time you run the program. $\endgroup$ – Brale May 15 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.