8
votes
Accepted
How can policy gradients be applied in the case of multiple continuous actions?
As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
- 3,817
6
votes
Why is the log probability replaced with the importance sampling in the loss function?
I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{...
- 10.3k
5
votes
Accepted
How is inequality 31 derived from equality 30 in lemma 2 of the "Trust Region Policy Optimization" paper?
We can start with equation (30):
$$
\bar{A}(s) = P(a \neq \tilde{a}) \mathbb{E}_{(a,\tilde{a})\sim(\pi,\tilde{\pi}|a\neq\tilde{a})} [A_\pi(s, \tilde{a}) - A_\pi(s, a)]
$$
Taking the absolute value ...
- 101
4
votes
Why is the log probability replaced with the importance sampling in the loss function?
For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am ...
- 81
3
votes
Accepted
Maximizing or Minimizing in Trust Region Policy Optimization?
The differences you have observed between the two different versions of the TRPO paper are due to different formalizations of the problem and the objective.
In the first version of the paper you ...
- 10.3k
2
votes
What makes TRPO an actor-critic method? Where is the critic?
TRPO still uses the advantage function, which computed using the critic (aka value function).
- 121
2
votes
Does importance sampling really improve sampling efficiency of TRPO or PPO?
The point of importance sampling is to use the same episode(s) to do multiple policy gradient updates. It will definitely increase sample efficiency over the strictly on-policy case, simply because in ...
- 1,251
2
votes
What is the difference between an on-policy distribution and state visitation frequency?
1.First of all. The on-policy distribution $\mu(s)$ is a probability distribution. So, obviously, it is different from state visitation frequency $\rho_\pi(s)$, since $\rho_\pi(s)$ is not normalized ...
- 2,524
1
vote
Is (log-)standard deviation learned in TRPO and PPO or fixed instead?
Using PPO from stable baselines 3, I see clearly that the variance of the actions are reducing in the environment I am using, where the optimal actions are static and not dependent on the state. This ...
1
vote
Accepted
Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?
I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the ...
- 380
1
vote
Are these two TRPO objective functions equivalent?
As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.
First, if you sample states according to their time ...
- 395
Only top scored, non community-wiki answers of a minimum length are eligible