1. As defined in the CPO paper, $d$ is the discounted future state distribution, roughly representing the weighted state distribution in the future. Consider infinite horizon with policy $\pi$, $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kP(s_k=s|\pi,s_0)\mu(s_0)\tag{1} $$ In the hierarchical case, we only consider the first $c$ steps, therefore $(1)$ needs to be truncated as $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^c \gamma^kP(s_k=s|\pi,s_0)\mu(s_0)\tag{2} $$ In the function form, it is $$ d_\pi=(1-\gamma)A_\pi\mu\tag{3} $$ But after truncation the sum of $d_\pi(s)$ over all states $s$ is not 1, therefore the normalizing term $(I-\gamma^cP_\pi^c)^{-1}$ needs to be multiplied, resulting in the finite horizon future state distribution $$ d_\pi=(1-\gamma)A_\pi(I-\gamma^cP_\pi^c)^{-1}\mu\tag{4} $$
2. The "every-$c$-step discounted state frequency" builds on $(4)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition matrix is $P_\pi^c$. Starting with $(4)$, replace $c$ with 1, we get the "every one step future state distribution" $$ d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{5} $$ Then replace $\gamma$ and $P_\pi$ in $(5)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution" $$ d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{6} $$

By the way, I read your blogpost on this paper. It's very helpful for me, thank you!

Update of the 1st part with respect to @Sherwin Chen's comment

Derivation of the normalizing term $(I-\gamma^cP_\pi^c)^{-1}$: Starting from $(3)$ (the function form), \begin{align} d_\pi&=(1-\gamma)A_\pi\mu\\ &=(1-\gamma)(\sum_{k=0}^{c-1}\gamma^kP_\pi^k)\mu\\ &=(1-\gamma)(1-\gamma P_\pi)^{-1}(I-\gamma^cP_\pi^c)\mu\\ &=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)(I-\gamma^cP_\pi^c)\mu \end{align} If the normalizing term is not multiplied, it gets us nowhere; however, if we multiply the normalizing term, use $\tilde d_\pi$ to denote the normalized $d_\pi$, then \begin{align} \tilde d_\pi&=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)(I-\gamma^cP_\pi^c)(I-\gamma^cP_\pi^c)^{-1}\mu\\ &=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)\mu \end{align} Explicitly write out the dependent variables: \begin{align} \tilde d_\pi(s)&=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^kP_\pi^k(s|s_0)\bigg]\mu(s_0) \end{align} \begin{align} \sum_{s\in S}\tilde d_\pi(s)&=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^k\sum_{s\in S}P_\pi^k(s|s_0)\bigg]\mu(s_0)\\ &=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^k\bigg]\mu(s_0)\\ &=\mu(s_0)=1 \end{align} as $\mu()$ is defined to be the Dirac delta centered at $s_0$.

Update: I rewrote the first part due to major mistake in the first version

Notice: The notation $P^k$ from Eq.$(20)$ and $(21)$ in the paper does not mean the kth power of some $P$. Instead, $P^k$ should be thought as the $k$ step transition probability of a non-homogeneous Markov chain.

1. According to the CPO paper, the discounted future state distribution is defined as $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb(s_k=s|\pi,s_0)\mu(s_0)\tag{1} $$ Consider function form. Let $Prob_{\pi,k}$ denote the $k$ step probability transition operator induced by $\pi$; here $\pi$ can be a hierarchical policy, $k$ can be larger than $c$. $$ d_\pi=(1-\gamma)\sum_{k=0}^\infty \gamma^kProb_{\pi,k}\mu\tag{2} $$ Now apply the similar definition as Eq.$(20)$ and $(21)$ in the paper, let $P^k_\pi$ denote the $k$ step transition probability of the non-homogeneous Markov chain induced by the low level policy, with $k$ smaller or equal to $c$. \begin{align} d_\pi&=(1-\gamma)\sum_{m=0}^\infty\gamma^{mc}\sum_{k=0}^{c-1}\gamma^kP^k_\pi(P^c_\pi)^m\mu\\ &=(1-\gamma)\sum_{k=0}^{c-1}\gamma^kP^k_\pi(\sum_{m=0}^\infty\gamma^{mc}(P^c_\pi)^m)\mu\\ &=(1-\gamma)A_\pi(I-\gamma^cP^c)^{-1}\mu\tag{3} \end{align} which is exactly the form of Eq.$(22)$ and $(23)$ in the paper, with $A_\pi$ defined similar as Eq.$(24)$ and $(25)$.
2. The "every-$c$-step discounted state frequency" builds on $(3)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition operator is $P_\pi^c$. Starting with $(3)$, replace $c$ with 1, we get the "every one step future state distribution" $$ d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{4} $$ Then replace $\gamma$ and $P_\pi$ in $(4)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution" $$ d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{5} $$ By the way, I read your blogpost on this paper. It's very helpful for me, thank you!

1. As defined in the CPO paper, $d$ is the discounted future state distribution, roughly representing the weighted state distribution in the future. Consider infinite horizon with policy $\pi$, $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kP(s_k=s|\pi)\tag{1} $$ In the hierarchical case, we only consider the first $c$ steps, therefore $(1)$ needs to be truncated as $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^c \gamma^kP(s_k=s|\pi)\tag{2} $$ In the function form, it is $$ d_\pi=(1-\gamma)A_\pi\mu\tag{3} $$ But after truncation the sum of $d_\pi(s)$ over all states $s$ is not 1, therefore the normalizing term $(I-\gamma^cP_\pi^c)^{-1}$ needs to be multiplied, resulting in the finite horizon future state distribution $$ d_\pi=(1-\gamma)A_\pi(I-\gamma^cP_\pi^c)^{-1}\mu\tag{4} $$ The meaning of $(4)$ is the same as $(1)$ (the weighted state distribution in the future), but the future has only finite horizon instead of infinite horizon.
2. The "every-$c$-step discounted state frequency" builds on $(4)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition matrix is $P_\pi^c$. Starting with $(4)$, replace $c$ with 1, we get the "every one step future state distribution" $$ d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{5} $$ Then replace $\gamma$ and $P_\pi$ in $(5)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution" $$ d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{6} $$

By the way, I read your blogpost on this paper. It's very helpful for me, thank you!

1. As defined in the CPO paper, $d$ is the discounted future state distribution, roughly representing the weighted state distribution in the future. Consider infinite horizon with policy $\pi$, $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^\infty \gamma^kP(s_k=s|\pi,s_0)\mu(s_0)\tag{1} $$ In the hierarchical case, we only consider the first $c$ steps, therefore $(1)$ needs to be truncated as $$ d_\pi(s)=(1-\gamma)\sum_{k=0}^c \gamma^kP(s_k=s|\pi,s_0)\mu(s_0)\tag{2} $$ In the function form, it is $$ d_\pi=(1-\gamma)A_\pi\mu\tag{3} $$ But after truncation the sum of $d_\pi(s)$ over all states $s$ is not 1, therefore the normalizing term $(I-\gamma^cP_\pi^c)^{-1}$ needs to be multiplied, resulting in the finite horizon future state distribution $$ d_\pi=(1-\gamma)A_\pi(I-\gamma^cP_\pi^c)^{-1}\mu\tag{4} $$
2. The "every-$c$-step discounted state frequency" builds on $(4)$, but it lumps the $c$ steps into one "high level" step where the discount factor is $\gamma^c$ and the transition matrix is $P_\pi^c$. Starting with $(4)$, replace $c$ with 1, we get the "every one step future state distribution" $$ d_\pi=(1-\gamma)(I-\gamma P_\pi)^{-1}\mu\tag{5} $$ Then replace $\gamma$ and $P_\pi$ in $(5)$ with $\gamma^c$ and $P_\pi^c$, we get the "every-$c$-step discounted state frequency", or "every-$c$-step future state distribution" $$ d_\pi^c=(1-\gamma^c)(I-\gamma^c P_\pi^c)^{-1}\mu\tag{6} $$

By the way, I read your blogpost on this paper. It's very helpful for me, thank you!

Update of the 1st part with respect to @Sherwin Chen's comment

Derivation of the normalizing term $(I-\gamma^cP_\pi^c)^{-1}$: Starting from $(3)$ (the function form), \begin{align} d_\pi&=(1-\gamma)A_\pi\mu\\ &=(1-\gamma)(\sum_{k=0}^{c-1}\gamma^kP_\pi^k)\mu\\ &=(1-\gamma)(1-\gamma P_\pi)^{-1}(I-\gamma^cP_\pi^c)\mu\\ &=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)(I-\gamma^cP_\pi^c)\mu \end{align} If the normalizing term is not multiplied, it gets us nowhere; however, if we multiply the normalizing term, use $\tilde d_\pi$ to denote the normalized $d_\pi$, then \begin{align} \tilde d_\pi&=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)(I-\gamma^cP_\pi^c)(I-\gamma^cP_\pi^c)^{-1}\mu\\ &=(1-\gamma)(\sum_{k=0}^{\infty}\gamma^kP_\pi^k)\mu \end{align} Explicitly write out the dependent variables: \begin{align} \tilde d_\pi(s)&=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^kP_\pi^k(s|s_0)\bigg]\mu(s_0) \end{align} \begin{align} \sum_{s\in S}\tilde d_\pi(s)&=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^k\sum_{s\in S}P_\pi^k(s|s_0)\bigg]\mu(s_0)\\ &=(1-\gamma)\bigg[\sum_{k=0}^{\infty}\gamma^k\bigg]\mu(s_0)\\ &=\mu(s_0)=1 \end{align} as $\mu()$ is defined to be the Dirac delta centered at $s_0$.