Questions tagged [trust-region-policy-optimization]

For questions about the Trust Region Policy Optimization (TRPO) algorithm, which was introduced in the paper "Trust Region Policy Optimization" (2015) by J. Schulman et al.

Filter by
Sorted by
Tagged with
0 votes
0 answers
32 views

CS 285 Prof Sergey Levine Lecture, Bounding Derivation for Reinforcement Learning (TRPO)

How can we derive the final result? I can understand the first line, but don't know how the absolute term in the summation is replaced with $2\epsilon t$. https://www.youtube.com/watch?v=LtAt5M_a0dI&...
user avatar
  • 101
0 votes
0 answers
12 views

Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?

I would like to shore up my interpretation of the concept of "batch size". It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the number of ...
user avatar
0 votes
0 answers
22 views

Can you apply TRPO to a problem involving a continuous state space and justify it theoretically?

I am currently reading and trying to understand the theory behind TRPO, i.e. sections 2 and 3 from the paper here. Ultimately, I want to apply PPO to do a (single) stock trading task using the FinRL ...
user avatar
0 votes
0 answers
30 views

How to calculate KL-divergence?

My problem setting is that KL-divergence is between two policies based on the data of one trajectory. As two ways to be shown below: $KL(\theta_{1}, \theta_{2}) = \sum\limits_{k=1}^{N}\pi_{\theta_{1}}...
user avatar
0 votes
0 answers
17 views

How can TRPO with constrained form allow larger update step?

There are two optimization forms of TRPO. One is that: \begin{equation}\max\limits_{\theta}[L_{\theta_{old}}(\theta) - CD^{max}_{KL}(\theta_{old}, \theta)]\end{equation} where $C = \frac{4\epsilon\...
user avatar
0 votes
1 answer
77 views

Does importance sampling really improve sampling efficiency of policy gradient methods such as TRPO or PPO?

Vanilla policy gradient has a loss function: $\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$ while in TRPO it is: $\mathcal{L}_{\pi_{\...
user avatar
0 votes
0 answers
29 views

Does it make sense to use the online network for making moves for both players in PPO for 2-player games?

I want to use PPO (or TRPO) to learn two-player games like 4 connect. Now, I'm thinking about whether or not it makes sense to use the online network for making moves for both players, in order to ...
user avatar
3 votes
0 answers
78 views

What is the difference between an on-policy distribution and state visitation frequency?

On-policy distribution is defined as follows in Sutton and Barto: On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization: $$\rho_{\pi}(s) = \sum_{t=0}^...
user avatar
2 votes
0 answers
268 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
user avatar
  • 705
1 vote
1 answer
64 views

Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?

I was going through the TRPO paper, and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand The ...
user avatar
  • 13
1 vote
0 answers
198 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
user avatar
  • 31
2 votes
1 answer
161 views

How can I implement the reward function for an 8-DOF robot arm with TRPO?

I need to get an 8-DOF (degrees of freedom) robot arm to move a specified point. I need to implement the TRPO RL code using OpenAI gym. I already have the gazebo environment. But I am unsure of how to ...
user avatar
3 votes
1 answer
160 views

Understanding proof of lemma 1 (policy improvement bound) of the "Trust Region Policy Optimization" paper

In the Trust Region Policy Optimization paper, in Lemma 1 of Appendix A, I did not quite understand the transition from (21) from (20). In going from (20) to (21), $A^\pi(s_t, a_t)$ is substituted ...
user avatar
  • 131
3 votes
1 answer
156 views

Are these two TRPO objective functions equivalent?

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
user avatar
  • 31
2 votes
0 answers
77 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
user avatar
2 votes
1 answer
163 views

Understanding lemma 2 of the "Trust Region Policy Optimization" paper

In the Trust Region Policy Optimization paper, in Lemma 2 of Appendix A, I did not quite understand deriving inequality (31) from (30), which is: $$\bar{A}(s) = P(a \neq \tilde{a} | s) \mathbb{E}_{(a,...
user avatar
6 votes
2 answers
1k views

Why is the log probability replaced with the importance sampling in the loss function?

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients $$L^...
user avatar
  • 96
3 votes
1 answer
65 views

Maximizing or Minimizing in Trust Region Policy Optimization?

I happened to discover that the v1 (19 Feb 2015) and the v5 (20 Apr 2017) versions of TRPO papers have two different conclusions. The Equation (15) in v1 is $\min_\theta$ while the Equation (14) in v2 ...
user avatar
  • 247
15 votes
1 answer
7k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...
user avatar