One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we discussed in Chapter 6, Deep Q-Network, when we talked about the correlation between samples, which breaks the independent and identically distributed (i.i.d) assumption, which is critical for Stochastic Gradient Descent (SDG) optimization. The negative consequence of such correlation is very high variance in gradients, which means that our training batch contains very similar examples, all of them pushing our network in the same direction. However, this may be totally the wrong direction in the global sense, as all those examples could be from one single lucky or unlucky episode. With our Deep Q-Network (DQN), we solved the issue by storing a large amount of previous states in the replay buffer and sampling our training batch from this buffer. If the buffer is large enough, the random sample from it is much better representation of the states distribution at large. Unfortunately, this solution won't work for PG methods, at most of them are on-policy, which means that we have to train on samples generated by our current policy, so, remembering old transitions is not possible anymore.
The above excerpt is from Maxim Lapan in the book Deep Reinforcement Learning Hands-on page 284.
How does being on-policy prevent us from using the replay buffer with the policy gradients? Can you explain to me mathematically why we can't use replay buffer with A3C for instance?