2022 Developer Survey is open! Take survey.
8 votes
Accepted

How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
user avatar
7 votes
Accepted

How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
user avatar
  • 9,326
6 votes

Why is the log probability replaced with the importance sampling in the loss function?

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story: In the PPO paper, after describing the standard policy gradient objective $L^{...
user avatar
  • 9,326
4 votes

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

Both implementations may be closer than you think. In short: PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
user avatar
  • 141
4 votes
Accepted

What is the effect of parallel environments in reinforcement learning?

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Yes they can make a difference. There are two ways improvement is seen: Collecting data from ...
user avatar
  • 23.2k
4 votes

Why is the log probability replaced with the importance sampling in the loss function?

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am ...
user avatar
  • 81
4 votes
Accepted

Why don't we decorrelate transitions for policy-based data?

We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like ...
user avatar
3 votes
Accepted

Do we use validation and test sets for training a reinforcement learning agent?

No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/...
user avatar
3 votes

What are the best hyper-parameters to tune in reinforcement learning?

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
user avatar
  • 512
3 votes

How is parallelism implemented in RL algorithms like PPO?

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
user avatar
  • 203
2 votes
Accepted

Why does the clipped surrogate objective work in Proximal Policy Optimization?

Ok, so I think I have a better understanding of this now. Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is ...
user avatar
  • 571
2 votes

How to implement a variable action space in Proximal Policy Optimization?

Change the action space at each step, depending on the internal_state. I assume this is nonsense. Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things ...
user avatar
2 votes

How do I calculate the policy in the Proximal Policy Optimization algorithm?

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\...
user avatar
2 votes

How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
user avatar
  • 1,773
2 votes
Accepted

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
user avatar
  • 23.2k
2 votes
Accepted

How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
user avatar
  • 131
2 votes

What are the best hyper-parameters to tune in reinforcement learning?

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor ...
user avatar
  • 746
2 votes

Why is training longer not better in reinforcement learning?

Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
user avatar
1 vote
Accepted

PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that $$ E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t) $$ Here we abuse ...
user avatar
  • 93
1 vote

Proximal Policy Optimization for continuous control problem

A different, variable reward structure might help. You could try a combination of airspeed, pitch, roll and whether it is hovering in the air or not in each timestep as a representation for the reward....
user avatar
  • 512
1 vote

Proximal Policy Optimization for continuous control problem

Training using only 20 timesteps at a time is far too small, especially when the goal will ultimately consist of episodes of length 6000. You definitely need to increase that substantially and that ...
user avatar
  • 746
1 vote
Accepted

How to define a continuous action distribution with a specific range for Reinforcement Learning?

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $...
user avatar
1 vote

Generation of 'new log probabilities' in continuous action space PPO

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...
user avatar
1 vote

Are these two TRPO objective functions equivalent?

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this. First, if you sample states according to their time ...
user avatar
1 vote

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
user avatar
1 vote

If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?

I had the same problem where the reward kept decreasing and started to search for answers in the forum. I let the model trained while I search. As the model trained, the reward started to increase. ...
user avatar
  • 11
1 vote

How is parallelism implemented in RL algorithms like PPO?

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts. They call their distributed training framework Rapid, which is also used in some ...
user avatar
  • 111
1 vote

Why does the clipped surrogate objective work in Proximal Policy Optimization?

I think @16Aghnar explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper: Engstrom et al., 2020, Implementation ...
user avatar
  • 226
1 vote

How to implement a variable action space in Proximal Policy Optimization?

Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible