# What is the difference between A2C and Q-Learning, and when to use one over the other?

I'm trying to get an accurate answer about the difference between A2C and Q-Learning. And when can we use each of them?

• Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
– Community Bot
Oct 11, 2022 at 0:05
• – Rob
Oct 11, 2022 at 0:16

The major difference between A2C and Q-Learning are what the algorithms learn. In A2C, and policy gradient algorithms in general, the policy is directly parameterised, i.e. we have $$\pi_\theta (a|s)$$. The parameters $$\theta$$ are typically optimised to maximise an objective that is a proxy for the expected returns $$\mathbb{E}_{\pi_\theta}\left[\sum_{i=0}^\infty \gamma^i r(s_i, a_i)\right]$$. Usually this proxy is the value function $$v_{\pi_\theta}(s)$$; the policy gradient theorem shows us that the derivative of this function wrt the policy parameters is $$\nabla_\theta v_{\pi_\theta}(s) = \mathbb{E}_{\pi_\theta}\left[G_t \nabla_\theta \log \pi_\theta (A | S)\right]$$.

In Q-Learning, we instead learn the Q-function: $$Q(s, a) = \mathbb{E}_\pi \left[r(s, a) + \gamma v_\pi(s') \right]$$. In a tabular MDP we can maintain exact estimates for every state-action pair, but if the state space is continuous/too large then we typically rely on function approximation, and so the Q-function would be parameterised in a similar way to the policy above. Now, as Q-Learning (in the tabular case) can be shown to converge to the Q-function under the optimal policy, i.e. the value of taking an action in a given state and thereafter following the optimal policy. We can then define a deterministic policy using this by $$\pi(s) = \arg\max_{a\in \mathcal{A}} Q_{\pi^*}(s, a)$$ where $$\pi^*$$ is the optimal policy and $$\mathcal{A}$$ is the action space of the MDP.

Note that in Actor-Critic methods we also learn a value function, similar to how the Q-function is learned in Q-learning.

Another major difference between the two algorithms is that Q-Learning is an off-policy algorithm; that is, the policy that the values are learnt for does not necessarily correspond to the policy that collects the data. A2C is an on-policy algorithm, so the data must correspond to data collected by the policy. This has disadvantages in data thirsty Deep Learning setups, but A3C can help overcome this by using many workers that share the parameters of the current policy to obtain many on-policy trajectories, so the amount of data being used would be similar to that of an off-policy algorithm.

Finally, I want to point out that policy gradient algorithms don't necessarily have to be on-policy -- Soft Actor Critic and Deterministic Policy Gradient are both examples of off-policy algorithms.

• Great answer. Another difference to consider is that A2C is an on-policy method, while Q-learning is off-policy. Oct 11, 2022 at 19:17
• Thanks, I might add that if I have time. I was trying to highlight the difference between value based and policy gradient, since policy gradient methods can also be off-policy (e.g. SAC, DDPG) Oct 12, 2022 at 7:16

Take a look at this blog: https://mpatacchiola.github.io/blog/2017/02/11/dissecting-reinforcement-learning-4.html

In a nutshell, the major difference between the two algorithms is: Q-learning consists of a critic only (to update state-action values) while A2C is composed of two networks: an actor (to take an action) and a critic (to evaluate and update state-action values). Major advantage over Q-learning is its computational efficiency, as far as I know.

• Got it! thank you!
– Hani
Oct 11, 2022 at 0:15