# Questions tagged [trust-region-policy-optimization]

For questions about the Trust Region Policy Optimization (TRPO) algorithm, which was introduced in the paper "Trust Region Policy Optimization" (2015) by J. Schulman et al.

6 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
251 views

### In lemma 1 of the TRPO paper, why isn't the expectation over $s'∼P(s'|s,a)$?

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I didn't quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted with ...
• 141
146 views

### How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
1 vote
573 views

### Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
• 31
36 views

### why PPO performs better than TRPO?

what is the logistic behind why PPO performs better than TRPO(at least in the PPO's paper results)? When I read both papers, it seems like PPO is just simpler version of TRPO, that it does not run ...
How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$. https://www.youtube.com/watch?v=LtAt5M_a0dI&...
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$. Is it possible to ...