3
$\begingroup$

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below):

algorithm

Why do the authors track this at every time step? Also, many blog posts and implementations leave this out (including I believe the OpenAI implementation on github).

Can someone explain explicitly how $\gamma_t$ is used in this algorithm?

Note: I understand the typical use of $\gamma$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.

$\endgroup$
4
$\begingroup$

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.

I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?

Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $\gamma_t$ values in the replay buffer altogether if it is known to be a constant $\gamma_t = \gamma$ for all $t$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $\gamma$.

$\endgroup$
  • 1
    $\begingroup$ Thanks for your answer. I do recall in Sutton & Barto the generalization to a variable gamma. But I rarely ever see it used (maybe I’ve never seen a real life implementation that wasn’t for research). I guess I forgot! $\endgroup$ – Hanzy May 31 at 12:11
  • 2
    $\begingroup$ @Hanzy You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things) $\endgroup$ – Dennis Soemers May 31 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.