In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ to measure the expected differential return $G_t^{diff} =\sum_{t=0}^\infty \left[R_{t} - r(\pi)\right]$ of policy $\pi$ in a state/action pair, or state, instead of its discounted return $G_t^{disc}\sum_{t=0}^\infty \gamma^t R_t$.
Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate $\bar{R} \rightarrow r(\pi)$ that starts at 0 and is then updated as $\bar{R}_{t +1} = \bar{R}_t + \beta \delta$, where $\beta$ is a learning rate and $\delta = R_{t + 1} - \bar{R}_t + \hat{q}(S_{t + 1}, A_{t + 1}) - \hat{q}(S_{t}, A_{t})$ is the action-value TD error, in a SARSA context where $A_{t+1}$ is sampled; I understand we might as well use the value function TD error $R_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_{t})$ if we are estimating that.
My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as $\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta R_{t+1}$ ?
