I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/):


Here, at every step in an episode we calculate the returns with involving the discount factors, such that the final gradient is like the following: $$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$

However, in Sutton´s book, the definition is like the following:


So, they explicity multiply each gradient term from each time step t, separately with the discount factor $\gamma^t$. Assuming they define $G_t$ similarly like $G_t = \sum_{k=t}^{T-1}\gamma^{k-t}r_k$, the obtained gradient will be:

$$\nabla J(\theta)=G_0\nabla_{\theta}log\pi_{\theta}(A_0|S_0) + \gamma G_1\nabla_{\theta}log\pi_{\theta}(A_1|S_1) + \gamma^2G_2\nabla_{\theta}log\pi_{\theta}(A_2|S_2) + \dots \gamma^{T-1}G_{T-1}\nabla_{\theta}log\pi_{\theta}(A_{T-1}|S_{T-1})$$

So, the gradient term in total will be different from the first one. Since in Sutton`s version they have used the discount factor reduntantly (it should be in $G_t$ already), it seems like the first version is the valid one to me, if this has not been done with a specific purpose. My question is, is there a small notational mistake in Sutton version of the algorithm, or am I missing something with the gradient formula here?


1 Answer 1


No, the difference between the gradients will be even greater, as in the second formulation the weights are updated each step, so the gradient is not a cumulative gradient, but a sort of 1-sample-SGD

In my opinion, the first one is the more accurate, as the following one does not consider the change in parameters (given $\theta' = \theta + \nabla_\theta L$, $\pi_\theta(A|S) \ne \pi_{\theta'}(A|S)$)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .