# In reinforcement learning, does the optimal value correspond to performing the best action in a given state?

I am confused about the definition of the optimal value ($$V^*$$) and optimal action-value (Q*) in reinforcement learning, so I need some clarification, because some blogs I read on Medium and GitHub are inconsistent with the literature.

Originally, I thought the optimal action value, $$Q^*$$, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.

And the optimal value, $$V^*$$, being the average $$Q$$ values in that state. Meaning that if you're in this state, the average "goodness" is this.

For example: If I am in a toy store and I can buy a pencil, yo-yo, or Lego.

Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50


And therefore my $$Q^* = 50$$

But my $$V^*$$ in this case is:

V* = -10 + 5 + 50 / 3 = 15


Representing no matter what action I take, the average future projected reward is $$15$$.

And for the advantage of learning, my baseline would be $$15$$. So anything less than $$0$$ is worse than average and anything above $$0$$ is better than average.

However, now I am reading about how $$V^*$$ actually assumes the optimal action in a given state, meaning $$V^*$$ would be 50 in the above case.

I am wondering which definition is correct.

I am wondering which definition is correct.

The asterisk * in both the definitions stands for "optimal" in the sense of "value when following the optimal policy"

So this one is correct:

$$V^*$$ actually assumes the optimal action in a given state, meaning $$V^*$$ would be $$50$$ in the above case

However, you have got the definition of Q slightly wrong.

I think this is because you are omitting the parameters.

The state value function uses the state as a parameter, $$V_{\pi}(s)$$, it returns the value of being in state $$s$$ and following a fixed policy $$\pi$$. The * is used to denote following an optimal policy.

The action value function has two parameters - a state and an action that is possible in that state, $$Q_{\pi}(s, a)$$, it returns the value of being in state $$s$$, taking action $$a$$ (regardless of whether it is the best action or not) and following the policy $$\pi$$ after that point.

And therefore my $$Q^* = 50$$
is wrong, or rather not meaningful, as you have not stated the parameters. You already list all the possible values of Q with the parameters. You could say $$\text{max}_a Q(\text{toy store}, a) = 50$$, or to choose the best action $$\pi(\text{toy store}) = \text{argmax}_a Q(\text{toy store}, a) = \text{Lego}$$