# Bellman equation and inverse matrix method

My problem: why the last formula do not contain information about time $$t$$? So if $$s^{\prime}=s$$, do we have $$v_{\pi}(s) = v_{\pi}(s^{\prime})$$? But this is not right I guess? If I am right, that they are not the same, considering Bellman equation for MRP $$\boldsymbol{v}=\boldsymbol{r}+\gamma \boldsymbol{P} \boldsymbol{v}$$. But why the left vector $$\boldsymbol{v}$$ and the right vector $$\boldsymbol{v}$$ are the same? The left one contains reward begin from $$t$$ but the right one contains reward begin from $$t+1$$, right?

In reinforcement learning, the apostrophe character $$(')$$ appended to a signal usually represents the signal at the next time step. For example, $$s'$$ is the state in the time step immediately after $$s$$. In your case, $$s$$ is the state at time step $$t$$; therefore, $$s'$$ is the state at time step $$t+1$$. Since the signals $$s$$, $$a$$, and $$r$$ do not have an appended apostrophe (or any other additional notation), they are assumed to be the state, action, and reward associated with the current time step $$t$$. To confirm, simply compare the last two equations and see how $$S_t$$ is replaced with $$s$$, $$A_t$$ with $$a$$, $$R_t$$ with $$r$$, and $$S_{t+1}$$ with $$s'$$. In summary, information about the associated time of the $$s$$, $$a$$, $$r$$, and $$s'$$ signals is implicitly encoded in their written representation.
• Thank you. I still have a problem about MRP Bellman equation in my comment, can you answer that? Many thanks. The key is I don't know why $v_\pi(s)$ and $v_\pi\left(s^{\prime}\right)$ can be regard as "same" in MRP, they related to rewards sum begin from $t$ and $t+1$ resptectively. Commented Dec 4, 2023 at 17:57