2
$\begingroup$

In the average reward setting we have:

$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$

$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$

How is the second equation derived from the first?

$\endgroup$
2
  • $\begingroup$ I think you are missing a factor of $\frac{1}{h}$ in the first equation. Otherwise the two equations are not describing the same thing $\endgroup$ – Neil Slater Oct 7 '20 at 19:17
  • $\begingroup$ at steady state distribution $E[R_t] = E[R_{t+1}] = E[R_{t+n}]$ If you substitute this into equation 1 you will end up with equation 2. And you are missing a factor $\frac{1}{h}$ in equation 1. $\endgroup$ – Swakshar Deb Oct 7 '20 at 20:29
2
$\begingroup$

We assume that our MDP is ergodic. Loosely speaking, this means that wherever the MDP starts (i.e. no matter which state we start in) or any actions the agent takes early on can only have a limited effect on the MDP and in the limit (as $t \rightarrow \infty$) the expectation of being in a given state depends only on the policy $\pi$ and the transition dynamics of the MDP.

This means that, eventually, $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$ for some large $t$. Therefore, as we take the average of our expected values of the rewards received for an infinitely long period of time, this will have converged due to what I just mentioned of $\mathbb{E}[R_t] = \mathbb{E}[R_{t+1}]$. To see why the two are equal, recall that the reward received is dependent on the current state and the action taken -- to better emphasise this I will briefly denote the reward at time step $t+1$ as $R(S_t, A_t)$. If we are in the steady state distribution, that is, the state distribution is now fixed, and our actions are still taken according to our policy, then the expected value of $R(S_t, A_t)$ will be the same for all future $t$ since neither the policy nor the state distribution are changing (recall that the average rewards are a way of evaluating a policy in the average-reward setting so for sure this does not change).

A way to think of this is that since we know that, eventually, $\mathbb{E}[R_t]$ will equal $\mathbb{E}[R_{t+1}]$, and so if we keep have an infinite number of these, the average of them will of course converge to the same value. Imagine if I gave you the sequence 1, 2, 3, 4, 4, 4, 4, ........, 4 and asked you to take the average - if we had an infinite amount of 4's then the average would of course be 4.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.