I have a simple question about model-free reinforcement. In a model I'm writing about, I want to know the value 'gain' we'd get for executing an action, relative to the current state. That is, what will I get if I moved from the current state $s$ taking action $a$.
The measure I want is:
$$G(s, a)=V(s^{\prime})-V(s)$$
where $s'$ is the state that I would transition to if the underlying MDP was deterministic. If the MDP has a stochastic transition function, the model I want is:
$$G(s, a)=\left[\sum_{s' \in S } P(s^{\prime} \mid a, s) V(s^{\prime})\right]-V(s)$$
In a model-free environment, we don't have $P(s' \mid a,s)$.
If we had a Q-function $Q(s,a)$, could we represent $G(s,a)$?
NOTE: This is not the same as an 'advantage function' as first proposed by Baird (Leemon C Baird. Reinforcement learning in continuous time: Advantage updating. In Proceedings of 1994 IEEE International Conference on Neural Networks, pages 448–2453. IEEE, 1994.), which means the advantage of actions relative to the optimal action. What I'm looking for is the gain of actions relative to the current state.
