2
$\begingroup$

In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):

$$ G_{t:t+1} \triangleq R_{t+1}+\gamma V_{t}(S_{t+1}) $$

where $V_t: \mathcal{S} \to \mathbb{R} $ is the estimate at $\textbf{time t}$ of $v_{\pi}$.

My question is this: Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$? Since we are at time $t+1$ where state is $S_{t+1}$, it seems more logical to have $V_{t+1}(S_{t+1})$.

$\endgroup$
4
  • $\begingroup$ Can you provide the source of this notation? Where did you find it? $\endgroup$
    – nbro
    Mar 11 at 8:42
  • $\begingroup$ @ bro, Sutton-Barto's RL book, page 143. second paragraph. $\endgroup$ Mar 11 at 11:01
  • $\begingroup$ Please, next time, for completeness, provide this info directly into your post. $\endgroup$
    – nbro
    Mar 11 at 16:21
  • 2
    $\begingroup$ I would imagine that here $V_t$ denotes your estimate of $V$ itself at time $t$. So even though we are estimating the value of the next step, it is only our estimate of $V$ at the $t$th iteration of the algorithm. $\endgroup$ Mar 11 at 16:28

1 Answer 1

1
$\begingroup$

TD-learning is based on bootstrapping. The TD target $R_{t+1} + \gamma V_t(S_{t+1})$ describes the immediate reward (random variable), $R_{t+1}$, plus the discounted estimated return (starting from the next state $S_{t+1}$, which is also a random variable), $V_t(S_{t+1})$. The reason why the discounted estimated return of the next state is calculated based on $V_t$ is that we do not know $V_{t+1}$ yet, that is at the current iteration of the algorithm.

What we want to do, is to find the optimal $V = V^*$, which would be the same for all timesteps. Then, $V^*_t = V^*_{t+1}$.

$\endgroup$
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .