In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:

$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$

and the discounted total reward $J(\pi)$ as:

$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$

I have two questions regarding these equations.

Question 1

Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.

I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.

I have read in this question that "$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$". I have confirmed that this is true, but I don't understand why.

I tested this with a simple example:

Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.


$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$


$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$

So indeed if we sum them and multiply by $(1-\gamma)$ we get:

$$(1-0.9)\cdot(4+6) = 1$$

Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?

Question 2

Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.

Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?

  • $\begingroup$ Please, next time, ask only one question per post, even if you have related questions. If you have more than 1 question, ask each of them in their separate post. $\endgroup$
    – nbro
    Mar 8 '21 at 10:25

Question 1

The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows

$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$

When you multiply by $1-\gamma$ you get

$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$

Which can be equivalently written as

$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$

Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities

Question 2

Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this

$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_ {a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$

The discounted return form is usually denoted

$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$

Where you notice the discounting terms aren't normalised which motivates cancelling it out

  • $\begingroup$ So from what I understand you are saying that $(1-\gamma)$ is used to normalize (i.e., get the sum of values to be equal to 1) and $(\frac{1}{1-\gamma})$ is used to cancel that normalization? $\endgroup$ Mar 4 '21 at 15:56
  • 1
    $\begingroup$ precisely that. $\endgroup$
    – quest ions
    Mar 4 '21 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.