# REINFORCE differentiation on sum or single value?

I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the gradient of the objective function is calculated as follows:

From what I understand $$\sum_{t=0}^{H}\nabla_{\theta}\log{\pi_{\theta}(a_{t}|s_{t})}$$ is the sum through the entire trajectory and $$\pi_{\theta}(a_{t}|s_{t})$$ is the policy of the agent at time step $$t$$. However in Suton's book the gradient objective is defined differently.

There is only $$\nabla \ln{\pi(A_t | S_t)}$$ at time step $$t$$ and no sum of all time steps. So does the algorithm not consider the policy for the whole trajectory when updating? Only a single-step policy?

Furthermore, there is $$\gamma^{t}$$ (discounted reward) term in the latter and not the former. What is the reason for that?

Hopefully, someone can help me clarify this.

• the site you learn from accumulates the gradient at each action taken in the episode via the sum and performs a single gradient ascent step for all these actions (this is analogous to taking the loss over a batch before performing a gradient update in supervised learning), whereas Sutton and Barto perform a gradient ascent step for every action taken (analogous to performing a gradient update for every single data point individually in supervised learning). Sep 3 at 22:16
• @norbertk thanks, I missed the bit of the question about the second discount term. OP, essentially the second discount term is needed because you also need to discount your state distribution. This is rarely done in practice but it is theoretically incorrect to omit it, and this paper shows without the second discount factor the gradient is not actually a gradient at all. Sep 4 at 8:54
• Hello. Welcome to AI SE. Rather than writing "Help with understanding the REINFORCE algorithm", please, just put your specific question in the title.
– nbro
Sep 6 at 11:10