# Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?

In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):

$$G_{t:t+1} \triangleq R_{t+1}+\gamma V_{t}(S_{t+1})$$

where $$V_t: \mathcal{S} \to \mathbb{R}$$ is the estimate at $$\textbf{time t}$$ of $$v_{\pi}$$.

My question is this: Why do we have $$t$$ as subscript in $$V$$ instead of $$t+1$$ in the expression of $$G_{t:t+1}$$? Since we are at time $$t+1$$ where state is $$S_{t+1}$$, it seems more logical to have $$V_{t+1}(S_{t+1})$$.

• Can you provide the source of this notation? Where did you find it?
– nbro
Mar 11 at 8:42
• @ bro, Sutton-Barto's RL book, page 143. second paragraph. Mar 11 at 11:01
• Please, next time, for completeness, provide this info directly into your post.
– nbro
Mar 11 at 16:21
• I would imagine that here $V_t$ denotes your estimate of $V$ itself at time $t$. So even though we are estimating the value of the next step, it is only our estimate of $V$ at the $t$th iteration of the algorithm. Mar 11 at 16:28

TD-learning is based on bootstrapping. The TD target $$R_{t+1} + \gamma V_t(S_{t+1})$$ describes the immediate reward (random variable), $$R_{t+1}$$, plus the discounted estimated return (starting from the next state $$S_{t+1}$$, which is also a random variable), $$V_t(S_{t+1})$$. The reason why the discounted estimated return of the next state is calculated based on $$V_t$$ is that we do not know $$V_{t+1}$$ yet, that is at the current iteration of the algorithm.
What we want to do, is to find the optimal $$V = V^*$$, which would be the same for all timesteps. Then, $$V^*_t = V^*_{t+1}$$.