It's a subtle issue.
If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) is off by a factor of gamma relative to the actor-critic pseudo-code for episodic problems in the Sutton and Barto book (p.332 of January 2019 edition of http://incompleteideas.net/book/the-book.html). The Sutton and Barto book has the extra "first" gamma as labeled in your picture. So, either the book or the A3C paper is wrong? Not really.
The key is on p. 199 of the Sutton and Barto book:
If there is discounting (gamma < 1) it
should be treated as a form of termination, which can be done simply by including
a factor of in the second term of (9.2).
The subtle issue is that there are two interpretations to the discounting factor gamma:
- A multiplicative factor that puts less weight on distant future rewards.
- A probability, 1 - gamma, that a simulated trajectory spuriously terminates, at any time step. This interpretation only makes sense for episodic cases, and not continuing cases.
- Just multiply the future rewards and related quantities (V or Q) in the future by gamma.
- Simulate some trajectories and randomly terminate (1 - gamma) of them at each time step. Terminated trajectories give no immediate or future rewards.
The two interpretations of gamma are valid. But choosing one or the other means you are tackling a different problem. The math is slightly different and you end up with an extra gamma multiplying $G \nabla\ln\pi(a|s)$ with the second interpretation.
For example, if you are at step t=2 and gamma = 0.9, the algorithm for the second interpretation is that the policy gradient is $\gamma^2 G \nabla\ln\pi(a|s)$ or $0.81 G \nabla\ln\pi(a|s)$. This term has 19% less gradient power than the t=0 term for the simple reason that 19% of simulated trajectories have died off by t=2.
With the first interpretation of gamma, there is no such 19% decay. The policy gradient is just $G \nabla\ln\pi(a|s)$ at t=2. But gamma is still present within $G$ to discount the future rewards.
You can choose whichever interpretation of gamma, but you have to be mindful of the consequences to the algorithm. I personally prefer to stick with interpretation 1 just because it's simpler. So I use the algorithm in the A3C paper, not the Sutton and Barto book.
Your question was about the REINFORCE algorithm, but I have been discussing actor-critic. You have the exact same issue related to the two gamma interpretations and the extra gamma in REINFORCE.