0
$\begingroup$

In state-value function estimation in reinforcement learning, I see the following notations: $V_T(S_t),\, V_{T-1}(S_t)$. What is the difference between them?

The context is given below.

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ Please give some context - what was being described when using this notation. A link to where this is written or citing the book and chapter would also be useful. All I can tell you otherwise is that $T$ is an index to multiple state value functions. Plus there is probably an algorithm relating different state value functions. I could guess what that is, but much better to see the context $\endgroup$ Commented Jan 17 at 8:31
  • $\begingroup$ @NeilSlater, I added the context. $\endgroup$ Commented Jan 21 at 18:26
  • $\begingroup$ Thanks. Please could you give a link to the original, as it is an image which will be difficult for some users to read. I don't see the notation $V_{T-1}(S_t)$ anywhere in the image you posted? Could you either post an image with the notation you are asking about, or correct the notation you want to know about in the main question. $\endgroup$ Commented Jan 21 at 19:38
  • $\begingroup$ @NeilSlater, sorry It was the wrong photo. I updated it. There is no link, I had to take a screenshot. $\endgroup$ Commented Jan 21 at 22:28
  • $\begingroup$ A reference to whichever book, course etc this is from then? With page num etc. There is clearly some published source you are referencing. $\endgroup$ Commented Jan 22 at 7:28

2 Answers 2

4
$\begingroup$

The difference between $V_T(S_t)$ and $V_{T-1}(S_t)$ is simply that $V_{T-1}(S_t)$ is the estimate of $V$ after $T-1$ updates. The notation is a bit clumsy since there are two elements of 'time' here -- one being how many updates we have done, which is the subscript of the value function, and the other being the time point in the MDP, which is the subscript of the state.

$\endgroup$
5
  • $\begingroup$ it was a wrong picture. Could you please update your answer based on the new picture? $\endgroup$ Commented Jan 21 at 22:52
  • 1
    $\begingroup$ This explanation still tracks with my reading of the new picture. $\endgroup$
    – foreverska
    Commented Jan 21 at 22:58
  • 2
    $\begingroup$ @foreverska and mine. $\endgroup$
    – David
    Commented Jan 21 at 22:58
  • 2
    $\begingroup$ I think that the previous image also had some incorrect/awkward notation. The author also includes a note that they are being a bit loose with notation. I would note that the notaton does work, sort of, with a single timeline. The value function at time step $T-1$ can be just "the previous value function", as the author doesn't refer any other index values, and updates only occur at the end of episodes. $\endgroup$ Commented Jan 22 at 7:48
  • 1
    $\begingroup$ @NeilSlater true, though it would be much clearer if rather than $T-1$ they used something simple like a dash (') to denote the new value function -- similar to how SGD is usually presented. $\endgroup$
    – David
    Commented Jan 22 at 11:30
-3
$\begingroup$

$VT(St)$ is the state value function. $\bar{v}_t$ is the average of $v_i$, where we make $v_i$ over a rolling period starting from time $t$. $VT(St)$ is a state-value function, for any given time $St$, it is not time invariant. Because of initialization, we can say that $VT(St') = \bar{v}_t = v_t$ if $t=St'$. In this sense, it is $\text{stationary}$. I write $VT−1(St)$ because I feel it is more close to the true equation ($V_t = \sum_i^n \frac{1}{n} RV_i^t$ is true), and $VT(St)$ is in fact a derivation over $V_t$ with using sample mean.

$\endgroup$
2
  • $\begingroup$ What's worth a downvote, simply a lack of relevant understanding? $\endgroup$ Commented Jan 17 at 11:09
  • 2
    $\begingroup$ There seem to be a lot of errors in this answer, as well as additional variables that you introduce but do not explain. As a result, it is confusing to read and hard to tell what you are trying to say, or whether it answers the question. Example of an error: "any given time $S_t$" - but $S_t$ is not a time, it is the state observed at time $t$. It also doesn't help that you are writing $VT$ and $VT - 1$ instead of $V_T$ and $V_{T-1}$ $\endgroup$ Commented Jan 17 at 12:32

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .