I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what will I get if I moved from the current state $s$ taking action $a$.

The measure I want is:

$$G(s, a)=V(s^{\prime})-V(s)$$

where $s'$ is the state that I would transition to if the underlying MDP was deterministic. If the MDP has a stochastic transition function, the model I want is:

$$G(s, a)=\left[\sum_{s' \in S } P(s^{\prime} \mid a, s) V(s^{\prime})\right]-V(s)$$

In a model-free environment, we don't have $P(s' \mid a,s)$.

If we had a Q-function $Q(s,a)$, could we represent $G(s,a)$?

NOTE: This is not the same as an 'advantage function' as first proposed by Baird (Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks, pages 448–2453. IEEE, 1994.), which means the advantage of actions relative to the optimal action. What I'm looking for is the gain of actions relative to the current state.

  • 1
    $\begingroup$ It would be a good idea to call your value something other than "advantage" and to use a different notation than $A(s,a)$. Your question will otherwise be confusing to any reader familiar with the normal use of the term (which you refer), which is well known in reinforcement learning. I suggest you change it using edit $\endgroup$ – Neil Slater Jun 2 at 7:22
  • $\begingroup$ I suggest you also don't use the letter $G$, given that it's used for the return. Use the letter $B$ or any less common letter in RL. $\endgroup$ – nbro Jun 2 at 11:36
  • $\begingroup$ B(s) is the baseline function in actor-critic, which is arguably more confusing than G. To me G(s,a) is fine because it is easily distinguished between G_t or G. $\endgroup$ – gibberblot Jun 2 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.