Questions tagged [trust-region-policy-optimization]
For questions about the Trust Region Policy Optimization (TRPO) algorithm, which was introduced in the paper "Trust Region Policy Optimization" (2015) by J. Schulman et al.
9
questions with no upvoted or accepted answers
3
votes
1
answer
714
views
Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
- 775
3
votes
1
answer
209
views
In lemma 1 of the TRPO paper, why isn't the expectation over $s'∼P(s'|s,a)$?
In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I didn't quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted with ...
- 131
2
votes
1
answer
288
views
How can I implement the reward function for an 8-DOF robot arm with TRPO?
I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
- 21
2
votes
0
answers
124
views
How does the TRPO surrogate loss account for the error in the policy?
In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated
An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
- 21
1
vote
0
answers
357
views
Why does PPO lead to a worse performance than TRPO in the same task?
I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
- 31
0
votes
0
answers
27
views
Very high dimensional optimization with large budget, requiring high quality solutions
What would be theoretically the best performing optimization algorithm(s) in this case?
Very high dimensional problem: 250-500 parameters
Goal is to obtain very high quality solutions, not just "...
0
votes
0
answers
36
views
Can TRPO use replay buffers?
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
Is it possible to ...
- 1
0
votes
0
answers
51
views
CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)
How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$.
https://www.youtube.com/watch?v=LtAt5M_a0dI&...
- 101
0
votes
0
answers
83
views
Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?
I would like to shore up my interpretation of the concept of "batch size".
It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the number of ...
