4
$\begingroup$

In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns.

I have also seen mention of discounted state distributions. It is mentioned on page 199 of the Sutton and Barto textbook that if there is discounting then (for the state distribution) it should be treated as a form of termination, and it is implied that this can be achieved by adding a factor of $\gamma$ to the state transition dynamics of the MDP, so that now we have

$$\mu(s) = \frac{\eta(s)}{\sum_{s'} \eta(s')}\;;$$ where $\eta(s) = h(s) + \sum_{\bar{s}} \eta(\bar{s})\sum_a \pi(a|\bar{s}) \gamma p(s|\bar{s}, a)$ and $h(s)$ is the probability of the episode beginning in state $s$.

In my opinion, the book kind of skips over this and it is not immediately clear to me why we need to discount our state distribution if we have discounting in the episode.

My intuition would suggest that it is because we usually take an expectation of the returns over the state distribution (and action/transition dynamics), but, if we are discounting the (future) rewards, then we should also discount the future states to give them less importance. In Sergey Levine's lectures he provides a brief aside that I think agrees with my intuition but in a rather unsatisfactory way -- he introduces the idea of a 'death state' that we transition into at each step with probability $1-\gamma$ but he does not really provide a rigorous enough justification for thinking of it this way (unless it is just a useful mental model and not supposed to be rigorous).

I am wondering whether someone can provide a more detailed explanation as to why we discount the state distribution.

$\endgroup$

2 Answers 2

1
$\begingroup$

Not an exhaustive answer, but perhaps this blog post by Alessio Russo may be helpful. In particular, he states how

There is an equivalence between using a discount factor and reaching a terminal state in a Markov Decision Process. [...] Therefore, we simply need to introduce, artificially, the possibility of terminating the trajectory with a certain probability 1-γ.

The topic is not 100% clear to me yet either, but I feel like it makes much more sense now why this "death state" is used as a model to replace the discounted state distribution.

$\endgroup$
0
$\begingroup$

If you have three possible next states from the current state, by adding the discount factor you are introducing a fourth state. It can be a terminal state or some other state that is hidden. The probability of entering that hidden state 1-γ. The probability of all other three initial possible next state is γ.

Suppose you are entering the termination state by probability 1-γ. Your policy is giving a 1-γ probability of dying, so your policy has to avoid that termination and has to make a better choice now i.e in the current state.

$\endgroup$
1
  • $\begingroup$ This is basically the same as the Sergey Levine description, which I don’t find useful (and is why I asked the question). $\endgroup$
    – David
    Nov 14, 2023 at 17:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .