19
$\begingroup$

As far as I understand, Q-learning and policy gradients (PG) are the two major approaches used to solve RL problems. While Q-learning aims to predict the reward of a certain action taken in a certain state, policy gradients directly predict the action itself.

However, both approaches appear identical to me, i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Is the difference in the way the loss is back-propagated?

$\endgroup$
18
$\begingroup$

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).

Both methods are theoretically driven by the Markov Decision Process construct, and as a result use similar notation and concepts. In addition, in simple solvable environments you should expect both methods to result in the same - or at least equivalent - optimal policies.

However, they are actually different internally. The most fundamental differences between the approaches is in how they approach action selection, both whilst learning, and as the output (the learned policy). In Q-learning, the goal is to learn a single deterministic action from a discrete set of actions by finding the maximum value. With policy gradients, and other direct policy searches, the goal is to learn a map from state to action, which can be stochastic, and works in continuous action spaces.

As a result, policy gradient methods can solve problems that value-based methods cannot:

  • Large and continuous action space. However, with value-based methods, this can still be approximated with discretisation - and this is not a bad choice, since the mapping function in policy gradient has to be some kind of approximator in practice.

  • Stochastic policies. A value-based method cannot solve an environment where the optimal policy is stochastic requiring specific probabilities, such as Scissor/Paper/Stone. That is because there are no trainable parameters in Q-learning that control probabilities of action, the problem formulation in TD learning assumes that a deterministic agent can be optimal.

However, value-based methods like Q-learning have some advantages too:

  • Simplicity. You can implement Q functions as simple discrete tables, and this gives some guarantees of convergence. There are no tabular versions of policy gradient, because you need a mapping function $p(a \mid s, \theta)$ which also must have a smooth gradient with respect to $\theta$.

  • Speed. TD learning methods that bootstrap are often much faster to learn a policy than methods which must purely sample from the environment in order to evaluate progress.

There are other reasons why you might care to use one or other approach:

  • You may want to know the predicted return whilst the process is running, to help other planning processes associated with the agent.

  • The state representation of the problem lends itself more easily to either a value function or a policy function. A value function may turn out to have very simple relationship to the state and the policy function very complex and hard to learn, or vice-versa.

Some state-of-the-art RL solvers actually use both approaches together, such as Actor-Critic. This combines strengths of value and policy gradient methods.

$\endgroup$
  • $\begingroup$ What do you mean when you say that actor-critic combines the strength of both methods? To my understanding, the actor evaluates the best action to take based on state, and the critic evaluates that state's value, then feeds reward to the actor. Treating them as a single "Policy" unit still looks like policy gradient to me. Why is this actually like Q-learning? $\endgroup$ – Gulzar Jan 25 at 20:04
  • 1
    $\begingroup$ @Guizar: The critic learns using a value-based method (e.g. Q-learning). So, overall, actor-critic is a combination of a value method and a policy gradient method, and it benefits from the combination. One notable improvement over "vanilla" PG is that gradients can be assessed on each step, instead of at the end of each episode. If you are looking for a more detailed answer on this subject you should ask a question on the site. $\endgroup$ – Neil Slater Jan 26 at 8:45
  • $\begingroup$ @Guizar: Actually scratch the (e.g. Q-learning) as I'm getting confused between advantage actor-critic (which adjusts the baseline to be based on action-values) and the critic which is usually a simpler state value. However, the rest my description is still the same, the critic is usually updated using value-based TD methods, of which Q learning is also an example. $\endgroup$ – Neil Slater Jan 26 at 10:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.