# Are batches useful for REINFORCE without strong episode cutoffs?

I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don't use batches. This leads me to ask, are batch updates of the network actually useful in this context?

Adding on, in my particular environment there's not a real meaningful cutoff for episodes as it's really set up for a sort of continuous play. As such, any n-length trajectory + rewards I collect is just as valid as another. For that reason, it would seem to mean that a longer episode/trajectory would serve the same purpose batches tend to in network updating.

Is it expected then that batches are not particularly worthwhile in the REINFORCE context, or is this just coincidence of the implementation I'm using? And is that answer amended if there are no meaningful episode cutoffs?

In REINFORCE, if you generated several episodes, and calculated the gradient over all transitions over all episodes, this would reduce the variance of the gradient compared to regular REINFORCE where we sample one episode at a time. You might know that when estimating the sample mean of a population, the variance decreases like $$1/n$$ where $$n$$ is the sample size. That's true here, for exactly the same reason: if you generated $$n$$ episodes per REINFORCE gradient, the variance will be $$1/n$$ what it is in normal REINFORCE.

If we choose some $$n$$ and also multiply the learning rate by $$n$$, we would expect both versions of REINFORCE to perform about the same in terms of average reward vs wall time and average reward vs number of episodes. But the one with higher $$n$$ does less gradient updates. In practice, you might be able to tune $$n$$ as a hyperparameter, but really you need to be using a better algorithm than REINFORCE if you care about performance at all.

• Just adding that the contrast you made against PPO in the other answer is helpful to establish the last point (better using a diff algorithm) and provides some intuitions for the effect on REINFORCE as well (in case other readers see this).
– Josh
Oct 3 at 17:37

Yes I see in the repo you link, in reinforce.py they only perform a gradient update once every episode. It sounds like what you're asking about is the difference between that reinforce style, and the more popular (and also more efficient) PPO type style. In that latter way, we have something like \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(size_training_set)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1}; \text{ save transition to memory} \\ & \qquad \text{ for m epochs}: \\ & \qquad \qquad \text{ calculate advantages} \\ & \qquad \qquad \text{ for k mini-batches}: \\ & \qquad \qquad \qquad \text{make mini batch from training_set}\text{ and do policy gradient update } \end{align*} There are some other details such as importance sampling, so I would recommend you can try another repo's code first.

The advantage of the PPO way is that we spend more time training on mini-batches and less time sampling the environment (which is slower), we can use each transition in multiple mini-batches, and we can generate more varied data to train on. Grouping together transitions from different times in different episodes might help remove harmful correlations. Also, theoretically the batch size shouldn't really matter, but in practice it's important, and we can't even tune that with the reinforce way.

• This is a very helpful read to get more information on PPO and shows what additional it offers as contrast, but I'm afraid I was asking explicitly in the REINFORCE like context. In my case, I imagine batches as sets of episodes (or trajectories). Anecdotally I've so far found this to not "improve" training but it does decrease the variance ever so slightly while requiring more iterations. Perhaps the benefit in my case is reduced variance (stabilization) and reduced number of weight updates (possible speed, but it's a bit slower on raw epoch count), though it's unclear.
– Josh
Oct 2 at 20:56
• Ok I probably didn't understand your question then. You're considering reinforce that generates, say, 4 episodes, and then does the "batch" over all transitions over all 4 episodes? That seems like a really weird thing to do to me.
– Taw
Oct 3 at 0:54
• Correct. What I would be interested in learning is why that might be weird, if there are better alternatives (that still are ~ within the REINFORCE territory), and better conceptual understanding of it all.
– Josh
Oct 3 at 14:21