# What's $\mathbb{V}[\gamma]$ in Return-based Scaling: Yet Another Normalisation Trick for Deep RL?

In the paper Return-based Scaling: Yet Another Normalisation Trick for Deep RL, $$\sigma$$, the scaling factor to normalize the TD error, is computed as:

$$\sigma^2 = \mathbb{V}[R] + \mathbb{V}[\gamma]\mathbb{E}[G^2]$$

What does the variance of $$\gamma$$ means? I believe $$\gamma$$ is the discount rate for the discounted return.

Note that, perhaps unconventionally, we treat $$\gamma$$ as a random variable here, because it is zero at the final step of an episode (even when it is constant throughout the episode).
Later, it is stated that $$\mathbb{V}[\gamma]$$ is calculated based on the entire set of observed data. So the distribution of $$\gamma$$ is I imagine, categorical with two possible values; $$\gamma$$ or $$0$$, with probability being normalised total steps vs. number of trajectories.
• So say there are 3 trajectories, so 3 times $\gamma=0$, and say each trajectory has 5 steps, so $4\times3$ times $\gamma$. $12\gamma$ and $3\times0$, is that it? Commented Jul 21, 2023 at 0:54
• @Sanyou having your same doubt, I just think that $\gamma \sim Be(p)$ where $p$ is just the ratio between the steps and final-step in your MDP (so say you have a trajectory of 5, gamma is 0 only in the last step), so say you gave 10-step long episode, $E[\gamma] = 0.9 \gamma + 0.1 \cdot 0 = 0.9 \gamma$, thus the variance is $V[\gamma] = (0.9 - \gamma)^2 * 0.9 + (0.9 - 0)^2 * 0.1$ Commented Nov 9, 2023 at 11:17