From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and old policy.
NPG is a modification to "vanilla" policy gradient algorithms that modifies the direction of the update using the Fisher information matrix such that the direction is in that of maximal improvement per unit of KL divergence between new and old policies.
Now, I've skipped a lot, but throughout most introductory material on these two algorithms, I see no mention of a critic (i.e. auxiliary evaluator value function), yet know that at least TRPO is considered actor-critic. Why is that? Where does the critic come in?