Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?

Question

In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):

$$ G_{t:t+1} \triangleq R_{t+1}+\gamma V_{t}(S_{t+1}) $$

where $V_t: \mathcal{S} \to \mathbb{R} $ is the estimate at $\textbf{time t}$ of $v_{\pi}$.

My question is this: Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$? Since we are at time $t+1$ where state is $S_{t+1}$, it seems more logical to have $V_{t+1}(S_{t+1})$.

Answer 1

TD-learning is based on bootstrapping. The TD target $R_{t+1} + \gamma V_t(S_{t+1})$ describes the immediate reward (random variable), $R_{t+1}$, plus the discounted estimated return (starting from the next state $S_{t+1}$, which is also a random variable), $V_t(S_{t+1})$. The reason why the discounted estimated return of the next state is calculated based on $V_t$ is that we do not know $V_{t+1}$ yet, that is at the current iteration of the algorithm.

What we want to do, is to find the optimal $V = V^*$, which would be the same for all timesteps. Then, $V^*_t = V^*_{t+1}$.