3
$\begingroup$

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ to measure the expected differential return $G_t^{diff} =\sum_{t=0}^\infty \left[R_{t} - r(\pi)\right]$ of policy $\pi$ in a state/action pair, or state, instead of its discounted return $G_t^{disc}\sum_{t=0}^\infty \gamma^t R_t$.

Of course, the average rate of reward for a given policy cannot be known in advance but only estimated, so the authors propose keeping a running estimate $\bar{R} \rightarrow r(\pi)$ that starts at 0 and is then updated as $\bar{R}_{t +1} = \bar{R}_t + \beta \delta$, where $\beta$ is a learning rate and $\delta = R_{t + 1} - \bar{R}_t + \hat{q}(S_{t + 1}, A_{t + 1}) - \hat{q}(S_{t}, A_{t})$ is the action-value TD error, in a SARSA context where $A_{t+1}$ is sampled; I understand we might as well use the value function TD error $R_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_{t})$ if we are estimating that.

My question is: why does the update rule for the average rate of reward estimate consider TD error, and why are we not updating it simply as $\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta R_{t+1}$ ?

$\endgroup$

1 Answer 1

3
$\begingroup$

Mystery solved thanks to Exercise 10.8 in the book. The reason is that we want the running mean to converge to the actual value of the average reward.

With $\bar{R}_{t + 1} = \bar{R}_t + \beta \delta$ with $\delta = {R}_{t + 1} - \bar{R}_t + \hat{v}(S_{t + 1}) - \hat{v}(S_t)$, assuming value function estimation converged I would get

$\begin{align} \delta &= R_{t + 1} - \bar{R}_t + v(S_{t + 1}) - v(S_{t}) \\ &= R_{t + 1} - \bar{R}_t + (R_{t + 2} - r(\pi^*) + R_{t + 3} - r(\pi^*) + ...) - (R_{t + 1} - r(\pi^*) + R_{t + 2} - r(\pi^*) + ...) \\ &= R_{t + 1} - \bar{R}_t - (R_{t + 1} - r(\pi^*)) \\ &= r(\pi^*) - \bar{R}_t \end{align} $

And the estimation will converge to the average reward of the optimal policy, being updated as $\bar{R}_{t + 1} = (1 - \beta) \bar{R}_t + \beta r(\pi^*)$.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .