Many later RL algorithms like PPO or Duelling DQN estimate the advantage. I am not very sure of how that really helps.

For instance, the actor loss for a simple actor critic algorithm is given by -

loss = -1 * policy.logprob(action) * advantage,

where advantage = returns - value (here value is estimated using a function approximator which is the critic).

I don't see why the loss function just couldn't be -

loss = -1 * policy.logprob(action) * value

This article states the following

Advantage learning and Q-learning learn equally quickly when used with a lookup table. Advantage learning can learn many orders of magnitude faster than Q-learning in some cases where a function approximator is used, even a linear function approximator. Specifically, if time steps are "small" in the sense that the state changes a very small amount on each time step, then advantage learning would be expected to learn much faster than Q-learning.

But I don't know why advantage learning is faster than q-learning.


1 Answer 1


So first, you are absolutely right that both are possible but using the advantage reduces the variance and therefore speeds up the learning. I'm going to explain this with the REINFORCE algorithm.

The Policy Gradient theorem provides us with a simple formula of the gradient of a parameterized state-value function \begin{equation} \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} q_{\pi}(S_{t}, a) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right] \end{equation} Where $\boldsymbol{\theta}$ is the parameter vector, $\pi(a | s, \boldsymbol{\theta})$ is the probability of the policy $\pi$ to choose action $a$ in the state $s$ while the policy is parameterized by $\boldsymbol{\theta}$.
The REINFORCE algorithm uses this proportionality and simplifies it even more, to \begin{equation} \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[G_{t} \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta})) \right] \end{equation} which is translates exactly (just remove the $\nabla$-operator from the right and the left hand-side) to the loss function you mentioned what you mentioned:
loss = -1 * policy.logprob(action) * value

The REINFORCE algorithm can be generalized by adding a baseline which leads to: \begin{equation} \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[(G_{t} - b(s)) \nabla_{\boldsymbol{\theta}}\text{ln}(\pi(A_{t}|S_{t},\boldsymbol{\theta})) \right] \end{equation} with a state-dependent baseline $b(s)$. Note that this baseline cannot be dependent on the state, otherwise the Policy Gradient theorem does not hold. Since the future rewards $G_{t}$ vary with the state, it showed that varying the baseline with the state led to better results, therefore a natural choice is the state value function $v(S_{t})$. The reason why this reduces the variance is, that the $G_{t}$ for REINFORCE and $G_{t} - b(s)$ for REINFORCE with baseline just changes the prefactor of the gradient. High gradients (removing the baseline) also leads to greedier learning but can also be more susceptible to errors. Think of it as the analogue to a (seperate) learning-rate (for each state) in classical deep learning.

Coming back to q-learning vs. advantage learning we can see this as applying the Policy Gradient theorem vs. applying the policy gradient theorem with the state value function as a baseline.

\begin{equation} \nabla J(\boldsymbol{\theta}) \propto \mathbb{E}_{\pi}\left[\sum_{a} (q_{\pi}(S_{t}, a) \color{green}{ - v_{\pi}(S_{t})}) \nabla_{\boldsymbol{\theta}} \pi(a|S_{t}, \boldsymbol{\theta}) \right] \end{equation}

As we know the prefactor to the gradient is the advantage: \begin{equation} A(s, a) = q(s, a) - v(s) \end{equation}

TLDR: Advantage learning is basically q-learning with a baseline, which reduces the variance and therefore speeds up learning.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .