0
$\begingroup$

as far as i understand there are 3 categories of Reinforcement algorithms:

  1. Value-based methods (like DQN or Sarsa)
  2. Policy-based methods (like REINFORCE)
  3. Actor-critic-based methods (like A2C)

To which of those categories does PPO (Proximal Policy Optimization) belong to? As the name suggest it is based on policy optimization but I think it also uses an actor-critic-based structure?

$\endgroup$

1 Answer 1

2
$\begingroup$

PPO belongs to both category 2 and 3:

  • It has an actor-critic structure, meaning that it learns the optimal policy via the actor network, and the value function via the critic network.
  • Moreover, differently from SAC for example, the actor is learned via policy gradient (not Q-learning as in SAC). Therefore it is also policy-based.

Update: defining value-based, policy-based, and actor-critic agents - I'll provide an intuitive rather than technical definition.

In general, a reinforcement learning agent can be equipped (let's say) with one or more components such as a (figure from David Silver's lecture 1):

enter image description here

  • Policy, representing the agent's behavior function typically mapping states to actions;
  • Value function, i.e., a function or model that predicts the expected future reward;
  • Model of the environment that predicts its next state.

Now, not all these components are required (e.g., the env's model) and some of them can be implicit (like the policy, which can be extracted from the value function.) According to which is the "main" component, we can categorize agents into:

  • Value-based, which need only to learn a value function (either the state-value or state-action one). For such agents learning the value function is enough since the policy can be "extracted" from it. For example, from a Q-function is very easy to derive a policy that predicts the action associated with the higher state-action value. You can also have a policy from the value function itself, but a one-step model of the environment is needed since the future is involved. TD-learning and Q-learning are both examples of value-based agents. Moreover, we consider value functions the state-value, $V(s)$, action-value (or state-action) function, $Q(s,a)$, and the advantage function, $A(s,a)$.
  • Policy-based methods that learn directly a policy instead of first approximating a value function. The policy can be deterministic (i.e., a mapping) or stochastic (a probability distribution.) In both cases, a popular approach is to learn the policy's parameters by following the (deterministic) policy gradient, which induces a further specialization of this class of agents. Another way is to use evolutionary strategies (ES), or optimization algorithms. Classical REINFORCE and DPG are example policy-based agents.
  • Actor-critic agents, which merge both value- and policy-based methods by learning two models: the actor (policy) and critic (value function). Now the critic can be found having various roles, from variance reduction serving as a baseline (e.g., in A2C, PPO) but also as a way to learn a policy (e.g., DDPG, SAC, TD3). The important thing is that we don't use the critic to derive an implicit policy.
$\endgroup$
11
  • $\begingroup$ Thanks for your answer Luca. So this means that there are also mixtures of the 3 basic RL algorithms (value-based methods, policy-based methods, and actor-critique-based methods)? I thought that every RL algorithm can only be assigned to one single category. Further is REINFORCE only a policy-based method and A2C only a actor-critique-based method or also mixtures? $\endgroup$
    – PeterBe
    Jan 5 at 8:39
  • $\begingroup$ But aren't actor-critic methods by definition also policy-based? I think that's not clear from this answer. Maybe a definition of policy-based and actor-critic methods is required here. $\endgroup$
    – nbro
    Jan 5 at 13:16
  • $\begingroup$ @nbro: Thanks for your comment. Actually I don't know. As far as I understand there are those 3 classes for RL algorithms. Apparently in REINFORCE there is no actor-critique structure while in PPO there is and both are policy-based methods. Actor-critique based methods combine value-based and policy-based methods as far as I understand $\endgroup$
    – PeterBe
    Jan 6 at 10:36
  • 1
    $\begingroup$ @PeterBe have a read at my latest update. Hope it helps. :) $\endgroup$ Jan 11 at 17:01
  • 1
    $\begingroup$ @PeterBe True: PPO is best classified as actor-critic, indeed, but it's also a policy-based method. I mean, this is a broader class than Actor-critic, which is more specific $\endgroup$ Jan 29 at 19:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .