3
$\begingroup$

it seems I am a little confused about the optimal value (V*) and optimal action-value (Q*) in reinforcement learning and just want some clarity because some blogs I read on Medium and GitHub are inconsistent with literature.

Originally, I thought the optimal action value, Q*, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.

And the optimal value, V*, being the average Q values in that state. Meaning that if you're in this state, the average "goodness" is this.

For example: If I am in a toy store and I can buy a pencil, yo-yo, or Lego.

Q(toy store, pencil) = -10

Q(toy store, yo-yo) = 5

Q(toy store, Lego) = 50

And therefore my Q* = 50

But my V* in this case is:

V* = -10 + 5 + 50 / 3 = 15

representing no matter what action I take, the average future projected reward is 15.

And for advantage learning, my baseline would be 15. So anything less than 0 is worse than average and anything above 0 is better than average.

However, now I am reading about how V* actually assumes the optimal action in a given state, meaning V* would be 50 in the above case.

I am wondering which definition is correct.

Thanks in advance!

$\endgroup$
3
$\begingroup$

I am wondering which definition is correct.

The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"

So this one is correct:

V* actually assumes the optimal action in a given state, meaning V* would be 50 in the above case

However, you have got the definition of Q slightly wrong.

I think this is because you are omitting the parameters.

The state value function uses the state as a parameter, $V_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The * is used to denote following an optimal policy.

The action value function has two parameters - a state and an action that is possible in that state, $Q_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is best action or not) and following a the policy $\pi$ after that point.

Your assertion in the question:

And therefore my Q* = 50

is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You could say $\text{max}_a Q(toy store, a) = 50$, or to choose the best action $\pi(toy store) = \text{argmax}_a Q(toy store, a) = \text{Lego}$

$\endgroup$
  • 1
    $\begingroup$ Ok I see. So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter. Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases? $\endgroup$ – Rui Nian Aug 24 '18 at 20:40
  • 1
    $\begingroup$ @RuiNian: Yes, the maximum Advantage when you already have an optimal policy is zero. $\endgroup$ – Neil Slater Aug 24 '18 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.