2
$\begingroup$

From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and old policy.

NPG is a modification to "vanilla" policy gradient algorithms that modifies the direction of the update using the Fisher information matrix such that the direction is in that of maximal improvement per unit of KL divergence between new and old policies.

Now, I've skipped a lot, but throughout most introductory material on these two algorithms, I see no mention of a critic (i.e. auxiliary evaluator value function), yet know that at least TRPO is considered actor-critic. Why is that? Where does the critic come in?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

TRPO still uses the advantage function, which computed using the critic (aka value function).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Oct 26, 2022 at 2:53

You must log in to answer this question.