In the appendix of the Constrained Policy Optimization (CPO) paper (Arxiv), the authors denote the discounted future state distribution $d^\pi$ as:
$$d^\pi(s) = (1-\gamma) \sum_{t=0}^\infty{\gamma^t P(s_t = s \vert \pi)}\tag1$$
and the discounted total reward $J(\pi)$ as:
$$J(\pi) = \frac{1}{1-\gamma} E_{\substack{s\sim d^\pi \\ a \sim \pi \\ s' \sim P}}[R(s,a,s')]\tag2$$
I have two questions regarding these equations.
Question 1
Intuitively, I understand that $d^\pi(s)$ returns the discounted probability of landing on state $s$ when executing policy $\pi$.
I understand that the summation part of $(1)$ results in values that are greater than $1$, and are, therefore, not fit for a probability distribution. But I do not understand why the value that results from this is multiplied by $(1-\gamma)$.
I have read in this question that "$(1−\gamma)$ normalizes all weights introduced by γ so that they are summed to $1$". I have confirmed that this is true, but I don't understand why.
I tested this with a simple example:
Suppose there is are only two states $s_A$ and $s_B$ and the probabilty of landing on $s_A$ is $0.4$ and on $s_B$ is $0.6$, independently of the previous state or action taken (therefore, independently of the policy $\pi$). Also suppose we set the maximum number of time steps $t_{max} = 1000$ (to make the equation easy to compute) and $\gamma = 0.9$.
Then:
$$d^\pi(s_A) = (1-0.9) \sum_{t=0}^{1000} 0.9^t \cdot 0.4 \approx (1-0.9) \cdot 4$$
and
$$d^\pi(s_B) \approx (1-0.9) \cdot 6$$
So indeed if we sum them and multiply by $(1-\gamma)$ we get:
$$(1-0.9)\cdot(4+6) = 1$$
Q: My question is why does multiplying by $(1-\gamma)$ normalize to $1$? And what does $(1-\gamma)$ represent in this context?
Question 2
Similarly, I can't understand the use of $\frac{1}{1-\gamma}$ in $(2)$.
Q: How does multiplying the expected value of the reward function by $\frac{1}{1-\gamma}$ result in the discounted reward, instead of multiplying by $\gamma$? What does $\frac{1}{1-\gamma}$ represent?