In state-value function estimation in reinforcement learning, I see the following notations: $V_T(S_t),\, V_{T-1}(S_t)$. What is the difference between them?
The context is given below.
In state-value function estimation in reinforcement learning, I see the following notations: $V_T(S_t),\, V_{T-1}(S_t)$. What is the difference between them?
The context is given below.
The difference between $V_T(S_t)$ and $V_{T-1}(S_t)$ is simply that $V_{T-1}(S_t)$ is the estimate of $V$ after $T-1$ updates. The notation is a bit clumsy since there are two elements of 'time' here -- one being how many updates we have done, which is the subscript of the value function, and the other being the time point in the MDP, which is the subscript of the state.
$VT(St)$ is the state value function. $\bar{v}_t$ is the average of $v_i$, where we make $v_i$ over a rolling period starting from time $t$. $VT(St)$ is a state-value function, for any given time $St$, it is not time invariant. Because of initialization, we can say that $VT(St') = \bar{v}_t = v_t$ if $t=St'$. In this sense, it is $\text{stationary}$. I write $VT−1(St)$ because I feel it is more close to the true equation ($V_t = \sum_i^n \frac{1}{n} RV_i^t$ is true), and $VT(St)$ is in fact a derivation over $V_t$ with using sample mean.