What is the intuition behind the TD(0) equation with average reward, and how is it derived?

Question

In chapter 10 of Sutton and Barto's book (2nd edition) is given the equation for TD(0) error with average reward (equation 10.10):

$$\delta_t = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, \mathbf{w}) - \hat{v}(S_{t}, \mathbf{w})$$

What is the intuition behind this equation? And how exactly is it derived?

Also, in chapter 13, section 6, is given the Actor-Critic algorithm, which uses the TD error. How can you use 1 error to update 3 distinct things - like the average reward, value function estimator (critic), and the policy function estimator (actor)?

Average Reward update rule: $\bar{R} \leftarrow \bar{R} + \alpha^{\bar{R}}\delta$

Critic weight update rule: $\mathbf{w} \leftarrow \mathbf{w} + \alpha^{\mathbf{w}}\delta\nabla \hat{v}(s,\mathbf{w})$

Actor weight update rule: $\mathbf{\theta} \leftarrow \mathbf{\theta} + \alpha^{\mathbf{\theta}}\delta\nabla ln \pi(A|S,\mathbf{\theta})$

Answer 1

This is simply from definition of return in average reward setting (look at equation $10.9$). The "standard" TD error is defined as \begin{equation} TD_{\text{error}} = R_{t+1} + V(S_{t+1}) - V(S_t) \end{equation} In average reward setting, average reward $r(\pi)$ is subtracted from reward at $t$, $R_t$, so TD error in this case is \begin{equation} TD_{\text{error}} = R_{t+1} - \bar R_{t+1} + V(S_{t+1}) - V(S_t) \end{equation} where $\bar R_{t+1}$ is estimate of $r(\pi)$.

You can use $\delta_t$ in all 3 updates because neither of these updates depend on each other. For example if you update $\mathbf w$ you don't use that then to update $\mathbf \theta$, or if you update $\bar R$ you don't use updated version to update $\mathbf w$ or $\mathbf \theta$ so you're not introducing additional bias. In each separate update you also don't have $\delta_t$ present multiple times so that you require multiple sampling per timestep to get the unbiased update.

Additionally this is semi-gradient algorithm, it uses bootstrapped estimate $V_{t+1}$ but it doesn't calculate full derivative with respect to it aswell, only with respect to $V_t$ so the algorithm is biased by default, but works well enough in practice for linear case.