# Questions tagged [trust-region-policy-optimization]

For questions about the Trust Region Policy Optimization (TRPO) algorithm, which was introduced in the paper "Trust Region Policy Optimization" (2015) by J. Schulman et al.

9 questions
Filter by
Sorted by
Tagged with
18 views

### Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
71 views

### How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
91 views

### Understanding proof of lemma 1 (policy improvement bound) of the “Trust Region Policy Optimization” paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
114 views

### Are these two TRPO objective functions equivalent?

In the TRPO paper, the objective to maximize is (equation 14) $$\mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right]$$ which ...
43 views

### How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
125 views

In the Trust Region Policy Optimization paper, in Lemma 2 of Appendix A, I did not quite understand deriving inequality (31) from (30), which is: $$\bar{A}(s) = P(a \neq \tilde{a} | s) \mathbb{E}_{(a,... 2answers 735 views ### Why is the log probability replaced with the importance sampling in the loss function? In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients$$L^...
I happened to discover that the v1 (19 Feb 2015) and the v5 (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min_\theta$ while the Equation (14) in v2 ...