# Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $$\gamma_t$$ in every state transition tuple (see line 6 in algorithm below):

Why do the authors track this at every time step? Also, many blog posts and implementations leave this out (including I believe the OpenAI implementation on github).

Can someone explain explicitly how $$\gamma_t$$ is used in this algorithm?

Note: I understand the typical use of $$\gamma$$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.

In some cases we may wish to have a discount factor $$\gamma_t$$ which depends on time $$t$$ (or depends on state $$s_t$$ and/or action $$a_t$$, leading to an indirect dependence on time $$t$$). Indeed we do not usually do this, but it does happen sometimes.
Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $$\gamma_t$$ values in the replay buffer altogether if it is known to be a constant $$\gamma_t = \gamma$$ for all $$t$$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $$\gamma$$.
• @Hanzy You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things) – Dennis Soemers May 31 '19 at 12:28