8
votes
Accepted
How can policy gradients be applied in the case of multiple continuous actions?
As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
- 3,557
7
votes
Accepted
How to implement a variable action space in Proximal Policy Optimization?
The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
- 9,326
6
votes
Why is the log probability replaced with the importance sampling in the loss function?
I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{...
- 9,326
4
votes
What are the pros and cons of using standard deviation or entropy for exploration in PPO?
Both implementations may be closer than you think.
In short:
PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
- 141
4
votes
Accepted
What is the effect of parallel environments in reinforcement learning?
Do parallel environments improve the agent's ability to learn or does it not really make a difference?
Yes they can make a difference. There are two ways improvement is seen:
Collecting data from ...
- 23.2k
4
votes
Why is the log probability replaced with the importance sampling in the loss function?
For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am ...
- 81
4
votes
Accepted
Why don't we decorrelate transitions for policy-based data?
We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like ...
- 1,918
3
votes
Accepted
Do we use validation and test sets for training a reinforcement learning agent?
No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/...
- 4,026
3
votes
What are the best hyper-parameters to tune in reinforcement learning?
Personally, I would choose the following two as the most important:
epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
- 512
3
votes
How is parallelism implemented in RL algorithms like PPO?
OpenAI have a post on that: https://openai.com/blog/openai-five/
They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
- 203
2
votes
Accepted
Why does the clipped surrogate objective work in Proximal Policy Optimization?
Ok, so I think I have a better understanding of this now.
Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is ...
- 571
2
votes
How to implement a variable action space in Proximal Policy Optimization?
Change the action space at each step, depending on the internal_state. I assume this is nonsense.
Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things ...
2
votes
How do I calculate the policy in the Proximal Policy Optimization algorithm?
You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\...
- 266
2
votes
How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?
Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
- 1,773
2
votes
Accepted
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
The answer to both your concerns is:
Add the previous action choice to the state representation.
It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
- 23.2k
2
votes
Accepted
How are continuous actions sampled (or generated) from the policy network in PPO?
As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
A common example for continuous spaces is the reparameterization trick, ...
- 131
2
votes
What are the best hyper-parameters to tune in reinforcement learning?
You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.
It suggests that discount factor ...
- 746
2
votes
Why is training longer not better in reinforcement learning?
Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
- 21
1
vote
Accepted
PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$
Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that
$$
E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)
$$
Here we abuse ...
- 93
1
vote
Proximal Policy Optimization for continuous control problem
A different, variable reward structure might help. You could try a combination of airspeed, pitch, roll and whether it is hovering in the air or not in each timestep as a representation for the reward....
- 512
1
vote
Proximal Policy Optimization for continuous control problem
Training using only 20 timesteps at a time is far too small, especially when the goal will ultimately consist of episodes of length 6000. You definitely need to increase that substantially and that ...
- 746
1
vote
Accepted
How to define a continuous action distribution with a specific range for Reinforcement Learning?
First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $...
- 4,026
1
vote
Generation of 'new log probabilities' in continuous action space PPO
The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...
- 522
1
vote
Are these two TRPO objective functions equivalent?
As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this.
First, if you sample states according to their time ...
- 395
1
vote
Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
- 161
1
vote
If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?
I had the same problem where the reward kept decreasing and started to search for answers in the forum.
I let the model trained while I search. As the model trained, the reward started to increase. ...
- 11
1
vote
How is parallelism implemented in RL algorithms like PPO?
The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts.
They call their distributed training framework Rapid, which is also used in some ...
- 111
1
vote
Why does the clipped surrogate objective work in Proximal Policy Optimization?
I think @16Aghnar explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper:
Engstrom et al., 2020, Implementation ...
- 226
1
vote
How to implement a variable action space in Proximal Policy Optimization?
Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position ...
- 11
Only top scored, non community-wiki answers of a minimum length are eligible