1
$\begingroup$

In the paper Return-based Scaling: Yet Another Normalisation Trick for Deep RL, $\sigma$, the scaling factor to normalize the TD error, is computed as:

$\sigma^2 = \mathbb{V}[R] + \mathbb{V}[\gamma]\mathbb{E}[G^2]$

What does the variance of $\gamma$ means? I believe $\gamma$ is the discount rate for the discounted return.

$\endgroup$

1 Answer 1

2
$\begingroup$

From the paper, note 3 of section 3.1.

Note that, perhaps unconventionally, we treat $\gamma$ as a random variable here, because it is zero at the final step of an episode (even when it is constant throughout the episode).

Later, it is stated that $\mathbb{V}[\gamma]$ is calculated based on the entire set of observed data. So the distribution of $\gamma$ is I imagine, categorical with two possible values; $\gamma$ or $0$, with probability being normalised total steps vs. number of trajectories.

$\endgroup$
2
  • $\begingroup$ So say there are 3 trajectories, so 3 times $\gamma=0$, and say each trajectory has 5 steps, so $4\times3$ times $\gamma$. $12\gamma$ and $3\times0$, is that it? $\endgroup$
    – Sanyou
    Commented Jul 21, 2023 at 0:54
  • 1
    $\begingroup$ @Sanyou having your same doubt, I just think that $\gamma \sim Be(p)$ where $p$ is just the ratio between the steps and final-step in your MDP (so say you have a trajectory of 5, gamma is 0 only in the last step), so say you gave 10-step long episode, $E[\gamma] = 0.9 \gamma + 0.1 \cdot 0 = 0.9 \gamma$, thus the variance is $V[\gamma] = (0.9 - \gamma)^2 * 0.9 + (0.9 - 0)^2 * 0.1$ $\endgroup$
    – Alberto
    Commented Nov 9, 2023 at 11:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .