What is the motivation for using Q-Learning in RL?

Question

In Spinning Up by OpenAI, it says the following regarding policy optimization methods and Q-Learning as ways of getting a good policy for RL.

Trade-offs Between Policy Optimization and Q-Learning. The primary strength of policy optimization methods is that they are principled, in the sense that you directly optimize for the thing you want. This tends to make them stable and reliable. By contrast, Q-learning methods only indirectly optimize for agent performance, by training $Q_{\theta}$ to satisfy a self-consistency equation. There are many failure modes for this kind of learning, so it tends to be less stable. But, Q-learning methods gain the advantage of being substantially more sample efficient when they do work, because they can reuse data more effectively than policy optimization techniques.

What I am wondering is the motivation behind Q-Learning in this sense; I understand that when it works, it can be nice getting better sample efficiency, but what I don't understand is why Q-Learning was even considered in the first place as a way to approximate the optimal policy. It seems counterintuitive to me to have something I want to optimize and then to not optimize it, but rather optimize something else.

In other words, why does Q-learning work when it does?

Answer 1

First, you asked why Q-Learning was considered. The reason is that it was a big revolution in the history of RL and the best method for a variety of problems at the time (1989, Watkins). Policy optimization was only introduced later by R. Williams in 1992.

There are some points where Q-Learning can have some advantages over policy optimization algorithms:

• tabular Q-learning is easy to implement and does not require any form of gradients. So for small environments (small in the sense that the all state-action pairs fit easily in memory or in a database) this might be a good fit.
• policy optimization models the policy while q-learning models the state-action value function (note that actor-critic methods can model both). The policy might be the easier function to model for some problems but not for all. So it depends on the problem which to use.
• Q-Learning is in general better understood theoretically (according to Sutton-Barto but this might not hold for deep Q-Learning)
• Q-Learning is an off-policy method, meaning that one can learn from another policy than the one that is being optimized which improves sample efficiency (note that there are soft actor critic and td3 which are also off-policy methods using policy gradients).
• policy gradient methods do require that the policy can be modeled by a differentiable function. Q-Learning does not have this requirement.

Why does Q-Learning work: In general reinforcement learning problems can be modeled by the Bellman equation: \begin{equation} q_{\pi}(s, a) = \mathbb{E}_{\pi}\left[\sum_{a}\pi(a|s)\sum_{r, s'}p(s', r|s,a)[r+\gamma v(s')]\right]. \end{equation} Problems like this can be tackled with various methods like dynamic programming, monte carlo methods or temporal difference methods, which combines the previous two. Q-Learning is a temporal difference method and there are theoretical proofs showing that it converges to optimality under certain conditions.

The problems of Q-Learning are more practical nature:

• the policy is indirectly modeled by the state-action value function which might not be ideal
• the policy can depend on hyperparamters for exploration which might not be tuned well or the actor policy might not visit crucial states often enough.