# Intuition behind $1-\gamma$ and $\frac{1}{1-\gamma}$ for calculating discounted future state distribution and discounted reward

In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $$d^\pi$$ as:

$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$

and the discounted total reward $$J(\pi)$$ as:

$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$

I have two questions regarding these equations.

Question 1

Intuitively, I understand that $$d^\pi(s)$$ returns the discounted probability of landing on state $$s$$ when executing policy $$\pi$$.

I understand that the summation part of $$(1)$$ results in values that are greater than $$1$$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $$(1-\gamma)$$.

I have read in this question that "$$(1−\gamma)$$ normalizes all weights introduced by γ so that they are summed to $$1$$". I have confirmed that this is true, but I don't understand why.

I tested this with a simple example:

Suppose there is are only two states $$s_A$$ and $$s_B$$ and the probabilty of landing on $$s_A$$ is $$0.4$$ and on $$s_B$$ is $$0.6$$, independently of the previous state or action taken (therefore, independently of the policy $$\pi$$). Also suppose we set the maximum number of time steps $$t_{max} = 1000$$ (to make the equation easy to compute) and $$\gamma = 0.9$$.

Then:

$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$

and

$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$

So indeed if we sum them and multiply by $$(1-\gamma)$$ we get:

$$(1-0.9)\cdot(4+6) = 1$$

Q: My question is why does multiplying by $$(1-\gamma)$$ normalize to $$1$$? And what does $$(1-\gamma)$$ represent in this context?

Question 2

Similarly, I can't understand the use of $$\frac{1}{1-\gamma}$$ in $$(2)$$.

Q: How does multiplying the expected value of the reward function by $$\frac{1}{1-\gamma}$$ result in the discounted reward, instead of multiplying by $$\gamma$$? What does $$\frac{1}{1-\gamma}$$ represent?

• Please, next time, ask only one question per post, even if you have related questions. If you have more than 1 question, ask each of them in their separate post. – nbro Mar 8 at 10:25

Question 1

The taylor expansion of $$\frac{1}{1-\gamma}$$ at $$\gamma= 0$$ is as follows

$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$

When you multiply by $$1-\gamma$$ you get

$$1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$

Which can be equivalently written as

$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$

Hence we can see that by multiplying the coefficients by $$(1-\gamma)$$ we get a weighted sum of transition probabilities

Question 2

Multiplying by $$(1-\lambda)$$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this

$$J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_ {a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$

The discounted return form is usually denoted

$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$

Where you notice the discounting terms aren't normalised which motivates cancelling it out

• So from what I understand you are saying that $(1-\gamma)$ is used to normalize (i.e., get the sum of values to be equal to 1) and $(\frac{1}{1-\gamma})$ is used to cancel that normalization? – jazzchipc Mar 4 at 15:56
• precisely that. – quest ions Mar 4 at 16:35