In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the discounted return is proportional to the undiscounted average reward $r(\pi)$ under the given policy.$^*$ They then advocate using average rewards rather than the usual returns of the discounted setting.

I've never encountered someone using average rewards (and no discounting) in the wild, though. Am I just ignorant of some use case, or is pretty much everyone sticking to discounting anyways?

$$r(\pi)=\sum_s \mu_\pi (s) \sum_a \pi(a|s) \sum_{s',r}p(s',r|s,a)r$$

$\mu_\pi$ is the stationary state distribution while following policy $\pi$.

$^*$Their proof did use the fact that the MDP was ergodic. I'm not sure how often that assumption holds in practice.

  • $\begingroup$ I have never actually seen a continuing task formulation used "in the wild". Possibly that explains something: episodic tasks are a good enough framework for most real world RL problems? $\endgroup$ – tahsmith Jul 12 '19 at 8:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.