Are batches useful for REINFORCE without strong episode cutoffs?

Question

I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don't use batches. This leads me to ask, are batch updates of the network actually useful in this context?

Adding on, in my particular environment there's not a real meaningful cutoff for episodes as it's really set up for a sort of continuous play. As such, any n-length trajectory + rewards I collect is just as valid as another. For that reason, it would seem to mean that a longer episode/trajectory would serve the same purpose batches tend to in network updating.

Is it expected then that batches are not particularly worthwhile in the REINFORCE context, or is this just coincidence of the implementation I'm using? And is that answer amended if there are no meaningful episode cutoffs?

Answer 1

In REINFORCE, if you generated several episodes, and calculated the gradient over all transitions over all episodes, this would reduce the variance of the gradient compared to regular REINFORCE where we sample one episode at a time. You might know that when estimating the sample mean of a population, the variance decreases like $1/n$ where $n$ is the sample size. That's true here, for exactly the same reason: if you generated $n$ episodes per REINFORCE gradient, the variance will be $1/n$ what it is in normal REINFORCE.

If we choose some $n$ and also multiply the learning rate by $n$, we would expect both versions of REINFORCE to perform about the same in terms of average reward vs wall time and average reward vs number of episodes. But the one with higher $n$ does less gradient updates. In practice, you might be able to tune $n$ as a hyperparameter, but really you need to be using a better algorithm than REINFORCE if you care about performance at all.

Answer 2

Yes I see in the repo you link, in reinforce.py they only perform a gradient update once every episode. It sounds like what you're asking about is the difference between that reinforce style, and the more popular (and also more efficient) PPO type style. In that latter way, we have something like \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(size_training_set)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1}; \text{ save transition to memory} \\ & \qquad \text{ for m epochs}: \\ & \qquad \qquad \text{ calculate advantages} \\ & \qquad \qquad \text{ for k mini-batches}: \\ & \qquad \qquad \qquad \text{make mini batch from training_set}\text{ and do policy gradient update } \end{align*} There are some other details such as importance sampling, so I would recommend you can try another repo's code first.

The advantage of the PPO way is that we spend more time training on mini-batches and less time sampling the environment (which is slower), we can use each transition in multiple mini-batches, and we can generate more varied data to train on. Grouping together transitions from different times in different episodes might help remove harmful correlations. Also, theoretically the batch size shouldn't really matter, but in practice it's important, and we can't even tune that with the reinforce way.