# Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $$\gamma$$ has no effect in continuing settings. They show (at least for one objective function) that the average of the discounted return is proportional to the undiscounted average reward $$r(\pi)$$ under the given policy.$$^*$$ They then advocate using average rewards rather than the usual returns of the discounted setting.

I've never encountered someone using average rewards (and no discounting) in the wild, though. Am I just ignorant of some use case, or is pretty much everyone sticking to discounting anyways?

$$r(\pi)=\sum_s \mu_\pi (s) \sum_a \pi(a|s) \sum_{s',r}p(s',r|s,a)r$$

$$\mu_\pi$$ is the stationary state distribution while following policy $$\pi$$.

$$^*$$Their proof did use the fact that the MDP was ergodic. I'm not sure how often that assumption holds in practice.

• I have never actually seen a continuing task formulation used "in the wild". Possibly that explains something: episodic tasks are a good enough framework for most real world RL problems? – tahsmith Jul 12 '19 at 8:35