2
$\begingroup$

enter image description here

My problem: why the last formula do not contain information about time $t$? So if $s^{\prime}=s$, do we have $v_{\pi}(s) = v_{\pi}(s^{\prime})$? But this is not right I guess? If I am right, that they are not the same, considering Bellman equation for MRP $\boldsymbol{v}=\boldsymbol{r}+\gamma \boldsymbol{P} \boldsymbol{v}$. But why the left vector $\boldsymbol{v}$ and the right vector $\boldsymbol{v}$ are the same? The left one contains reward begin from $t$ but the right one contains reward begin from $t+1$, right?

$\endgroup$
0

1 Answer 1

4
$\begingroup$

In reinforcement learning, the apostrophe character $(')$ appended to a signal usually represents the signal at the next time step. For example, $s'$ is the state in the time step immediately after $s$. In your case, $s$ is the state at time step $t$; therefore, $s'$ is the state at time step $t+1$. Since the signals $s$, $a$, and $r$ do not have an appended apostrophe (or any other additional notation), they are assumed to be the state, action, and reward associated with the current time step $t$. To confirm, simply compare the last two equations and see how $S_t$ is replaced with $s$, $A_t$ with $a$, $R_t$ with $r$, and $S_{t+1}$ with $s'$. In summary, information about the associated time of the $s$, $a$, $r$, and $s'$ signals is implicitly encoded in their written representation.

$\endgroup$
4
  • $\begingroup$ Thank you. I still have a problem about MRP Bellman equation in my comment, can you answer that? Many thanks. The key is I don't know why $v_\pi(s)$ and $v_\pi\left(s^{\prime}\right)$ can be regard as "same" in MRP, they related to rewards sum begin from $t$ and $t+1$ resptectively. $\endgroup$
    – Goldhand
    Dec 4, 2023 at 17:57
  • $\begingroup$ If possible, please edit the post to include your question inside the main body of the post. Also, note that this website has a very specific rule that each post can only have one question. If you feel that the question in your comment is separate from the question in the post, simply make another post with that second question. Thank you for posting and we look forward to more of your questions in the near future! $\endgroup$
    – DeepQZero
    Dec 4, 2023 at 18:01
  • $\begingroup$ Thank you. I've edited my problem. Actually they're related problems. Hope you can answer it if you like. $\endgroup$
    – Goldhand
    Dec 4, 2023 at 18:13
  • $\begingroup$ @DeepQZero Great! I like your answer. I voted for you! $\endgroup$ Dec 5, 2023 at 22:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .