I'm implementing PPO myself strictly follow the steps:
- sample transitions
- randomly shuffle the sampled transitions
- compute gradients and update networks using the sampled transitions
- drop transitions and repeat the above steps
I observe a strange phenomenon that randomly shuffling transitions makes the algorithm perform significantly worse than keeping it as it is. This is very strange to me. To my best understanding, neural networks perform badly when the input data are correlated. To decorrelate transitions, algorithms like DQN introduce replay buffer and randomly sample from it. But this seems not the same story to policy-based methods. I'm wondering why policy-based methods do not require to decorrelate the input data?