# Tag Info

It seems that you are getting confused between the definition of a Q-value and the update rule used to obtain these Q-values. Remember that to simply obtain an optimal Q-value for a given state-action pair we can evaluate $$Q(s, a) = r + \gamma \max_{a'} Q(s', a)\;;$$ where $s'$ is the state we transitioned into (note that this only holds when obtaining the ...
Your equations all look correct to me. It is not possible to solve the linear equation for state values in the vector $V$ without knowing the policy. There are ways of working with MDPs, through sampling of actions, state transitions and rewards, where it is possible to estimate value functions without knowing either $\pi(a|s)$ or $P^{a}_{ss'}$. For instance,...