2
$\begingroup$

One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?

$\endgroup$
1
  • $\begingroup$ This seems to be a duplicate of this. $\endgroup$
    – nbro
    Nov 7 '20 at 14:31
2
$\begingroup$

You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment $p(s',r|s,a)$ - the probability of receiving reward $r$ and arriving in state $s'$ when starting in state $s$ and taking action $a$.

This looks like:

$$\pi^*(s) = \text{argmax}_a [\sum_{s',r} p(s',r|s,a)(r + \gamma v^*(s'))]$$

There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.

Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.