In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns.
I have also seen mention of discounted state distributions. It is mentioned on page 199 of the Sutton and Barto textbook that if there is discounting then (for the state distribution) it should be treated as a form of termination, and it is implied that this can be achieved by adding a factor of $\gamma$ to the state transition dynamics of the MDP, so that now we have
$$\mu(s) = \frac{\eta(s)}{\sum_{s'} \eta(s')}\;;$$ where $\eta(s) = h(s) + \sum_{\bar{s}} \eta(\bar{s})\sum_a \pi(a|\bar{s}) \gamma p(s|\bar{s}, a)$ and $h(s)$ is the probability of the episode beginning in state $s$.
In my opinion, the book kind of skips over this and it is not immediately clear to me why we need to discount our state distribution if we have discounting in the episode.
My intuition would suggest that it is because we usually take an expectation of the returns over the state distribution (and action/transition dynamics), but, if we are discounting the (future) rewards, then we should also discount the future states to give them less importance. In Sergey Levine's lectures he provides a brief aside that I think agrees with my intuition but in a rather unsatisfactory way -- he introduces the idea of a 'death state' that we transition into at each step with probability $1-\gamma$ but he does not really provide a rigorous enough justification for thinking of it this way (unless it is just a useful mental model and not supposed to be rigorous).
I am wondering whether someone can provide a more detailed explanation as to why we discount the state distribution.
