# Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $$r(\pi)$$ to define the quality of a policy, re-defining action-value function $$q_\pi(s,a)$$ and value function $$v_\pi(s)$$ to measure the expected differential return $$G_t^{diff} =\sum_{t=0}^\infty \left[R_{t} - r(\pi)\right]$$ of policy $$\pi$$ in a state/action pair, or state, instead of its discounted return $$G_t^{disc}\sum_{t=0}^\infty \gamma^t R_t$$.

Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate $$\bar{R} \rightarrow r(\pi)$$ that starts at 0 and is then updated as $$\bar{R}_{t +1} = \bar{R}_t + \beta \delta$$, where $$\beta$$ is a learning rate and $$\delta = R_{t + 1} - \bar{R}_t + \hat{q}(S_{t + 1}, A_{t + 1}) - \hat{q}(S_{t}, A_{t})$$ is the action-value TD error, in a SARSA context where $$A_{t+1}$$ is sampled; I understand we might as well use the value function TD error $$R_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_{t})$$ if we are estimating that.

My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as $$\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta R_{t+1}$$ ?

## 1 Answer

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.

With $$\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$$ with $$\delta = {R}_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_t)$$, assuming value function estimation converged I would get

\begin{align} \delta &= R_{t + 1} - \bar{R}_t + v(S_{t + 1}) - v(S_{t}) \\ &= R_{t + 1} - \bar{R}_t + (R_{t + 2} - r(\pi^*) + R_{t + 3} - r(\pi^*) + ...) - (R_{t + 1} - r(\pi^*) + R_{t + 2} - r(\pi^*) + ...) \\ &= R_{t + 1} - \bar{R}_t - (R_{t + 1} - r(\pi^*)) \\ &= r(\pi^*) - \bar{R}_t \end{align}

And the estimation will converge to the average reward of the optimal policy, being updated as $$\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta r(\pi^*)$$.