3
$\begingroup$

In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:

$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$

and the discounted total reward $J(\pi)$ as:

$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$

I have two questions regarding these equations.

Question 1

Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.

I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.

I have read in this question that "$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$". I have confirmed that this is true, but I don't understand why.

I tested this with a simple example:

Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.

Then:

$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$

and

$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$

So indeed if we sum them and multiply by $(1-\gamma)$ we get:

$$(1-0.9)\cdot(4+6) = 1$$

Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?

Question 2

Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.

Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?

$\endgroup$
1
  • $\begingroup$ Please, next time, ask only one question per post, even if you have related questions. If you have more than 1 question, ask each of them in their separate post. $\endgroup$
    – nbro
    Mar 8, 2021 at 10:25

1 Answer 1

3
$\begingroup$

Question 1

The taylor expansion of $\frac{1}{1-\gamma}$ at $\gamma= 0$ is as follows

$$\frac{1}{1-\gamma} = 1 + \gamma + \gamma^2 + \dots$$

When you multiply by $1-\gamma$ you get

$$ 1 = (1-\gamma)(1 + \gamma + \gamma^2 + \dots)$$

Which can be equivalently written as

$$1 = (1-\gamma)\sum_\limits{i=0}^{\infty}\gamma^i$$

Hence we can see that by multiplying the coefficients by $(1-\gamma)$ we get a weighted sum of transition probabilities


Question 2

Multiplying by $(1-\lambda)$ is to cancel out the normalisation term which was included in the definition of the discounted distribution of the states. Partially expanding out your expressions for the discounted reward you get this

$$ J(\pi) = \frac{1}{1-\gamma}\sum_\limits{s}E_ {a\sim\pi,s'\sim P}[R(s,a,s')]\cdot\left((1-\gamma)\sum_\limits{t=0}^{\infty}\gamma^tP(s_t=s|\pi)\right)$$

The discounted return form is usually denoted

$$J(\pi) = E[\sum\gamma^iR_i(s,a,s')]$$

Where you notice the discounting terms aren't normalised which motivates cancelling it out

$\endgroup$
2
  • $\begingroup$ So from what I understand you are saying that $(1-\gamma)$ is used to normalize (i.e., get the sum of values to be equal to 1) and $(\frac{1}{1-\gamma})$ is used to cancel that normalization? $\endgroup$ Mar 4, 2021 at 15:56
  • 1
    $\begingroup$ precisely that. $\endgroup$
    – quest ions
    Mar 4, 2021 at 16:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .