2
$\begingroup$

One can easily retrieve the optimal policy from the action value function but how about obtaining it from the state value function?

$\endgroup$
1
  • $\begingroup$ This seems to be a duplicate of this and this. $\endgroup$
    – nbro
    Nov 7, 2020 at 14:31

1 Answer 1

2
$\begingroup$

You can obtain the optimal policy from the optimal state value function if you also have the state transition and reward model for the environment $p(s',r|s,a)$ - the probability of receiving reward $r$ and arriving in state $s'$ when starting in state $s$ and taking action $a$.

This looks like:

$$\pi^*(s) = \text{argmax}_a [\sum_{s',r} p(s',r|s,a)(r + \gamma v^*(s'))]$$

There are variations of this function, depending on how you represent knowledge of the environment. For instance, you don't actually need the full distribution model for reward, an expected reward function and separate distribution model for state transition rules would also work.

Without at least an approximate model of the environment, you cannot derive a policy from state values. If all you have is state values, then to pick an optimal action, you absolutely need the ability to look ahead a time step at what the next state might be for each action choice.

$\endgroup$
1
  • $\begingroup$ How would this extend to two player (and more than three or more player) games, say just adversarial zero-sum ones, assuming reward and transition probability is known? Specifically, how does the argmax work, since the next state corresponds to the next player? Is the state input to the value function ordered to be from the perspective of the current player? $\endgroup$
    – thinkski
    Jul 31 at 0:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .