it seems I am a little confused about the optimal value (V*) and optimal action-value (Q*) in reinforcement learning and just want some clarity because some blogs I read on Medium and GitHub are inconsistent with literature.
Originally, I thought the optimal action value, Q*, represents you performing the action that maximizes your current reward, and then acting optimally thereafter.
And the optimal value, V*, being the average Q values in that state. Meaning that if you're in this state, the average "goodness" is this.
For example: If I am in a toy store and I can buy a pencil, yo-yo, or Lego.
Q(toy store, pencil) = -10
Q(toy store, yo-yo) = 5
Q(toy store, Lego) = 50
And therefore my Q* = 50
But my V* in this case is:
V* = -10 + 5 + 50 / 3 = 15
representing no matter what action I take, the average future projected reward is 15.
And for advantage learning, my baseline would be 15. So anything less than 0 is worse than average and anything above 0 is better than average.
However, now I am reading about how V* actually assumes the optimal action in a given state, meaning V* would be 50 in the above case.
I am wondering which definition is correct.
Thanks in advance!