3
$\begingroup$

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below):

algorithm

Why do the authors track this at every time step? Also, many blog posts and implementations leave this out (including I believe the OpenAI implementation on github).

Can someone explain explicitly how $\gamma_t$ is used in this algorithm?

Note: I understand the typical use of $\gamma$ as a discount factor. But typically gamma remains fixed. Which is why I’m curious as to the need to track it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do not usually do this, but it does happen sometimes.

I guess that, from a theoretical point of view, it was very easy of the authors to make their algorithm more flexible/general and also support this (somewhat rare) case of time-varying discount factor. If it had been very complicated for them to support this option, they may have chosen not to; but if it's trivial to do so, well, why not?

Practical implementations will often indeed ignore that possibility if they're not using it, and can avoid including $\gamma_t$ values in the replay buffer altogether if it is known to be a constant $\gamma_t = \gamma$ for all $t$. As far as I can see, in the experiments discussed in this paper they also only used a fixed, constant $\gamma$.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer. I do recall in Sutton & Barto the generalization to a variable gamma. But I rarely ever see it used (maybe I’ve never seen a real life implementation that wasn’t for research). I guess I forgot! $\endgroup$
    – Hanzy
    May 31, 2019 at 12:11
  • 2
    $\begingroup$ @Hanzy You may also sometimes see similar things in papers for variable $\lambda$ (in Sarsa($\lambda$) and other TD($\lambda$) algorithms) and/or variable learning rates $\alpha$ (although I suppose having time-varying, in particular decreasing, learning rates is quite a bit more common than variable versions of the other things) $\endgroup$
    – Dennis Soemers ♦
    May 31, 2019 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .