3
$\begingroup$

I am confused about the definition of the optimal value ($V^*$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.

Originally, I thought the optimal action value, $Q^*$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.

And the optimal value, $V^*$, being the average $Q$ values in that state. Meaning that if you're in this state, the average "goodness" is this.

For example: If I am in a toy store and I can buy a pencil, yo-yo, or Lego.

Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50

And therefore my $Q^* = 50$

But my $V^*$ in this case is:

V* = -10 + 5 + 50 / 3 = 15

Representing no matter what action I take, the average future projected reward is $15$.

And for the advantage of learning, my baseline would be $15$. So anything less than $0$ is worse than average and anything above $0$ is better than average.

However, now I am reading about how $V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be 50 in the above case.

I am wondering which definition is correct.

$\endgroup$
3
$\begingroup$

I am wondering which definition is correct.

The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"

So this one is correct:

$V^*$ actually assumes the optimal action in a given state, meaning $V^*$ would be $50$ in the above case

However, you have got the definition of Q slightly wrong.

I think this is because you are omitting the parameters.

The state value function uses the state as a parameter, $V_{\pi}(s)$, it returns the value of being in state $s$ and following a fixed policy $\pi$. The * is used to denote following an optimal policy.

The action value function has two parameters - a state and an action that is possible in that state, $Q_{\pi}(s, a)$, it returns the value of being in state $s$, taking action $a$ (regardless of whether it is the best action or not) and following the policy $\pi$ after that point.

Your assertion in the question:

And therefore my $Q^* = 50$

is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You could say $\text{max}_a Q(\text{toy store}, a) = 50$, or to choose the best action $\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a) = \text{Lego}$

$\endgroup$
2
  • 1
    $\begingroup$ Ok I see. So V* assumes the agent takes the best possible action in the state, and then follows the optimal policy thereafter. Does that also mean the baseline at each state should be V* and that A(s, a) = 0 for the optimal case (Q(s,a) - V*), and always negative for all other cases? $\endgroup$
    – Rui Nian
    Aug 24 '18 at 20:40
  • 1
    $\begingroup$ @RuiNian: Yes, the maximum Advantage when you already have an optimal policy is zero. $\endgroup$ Aug 24 '18 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.