2022 Developer Survey is open! Take survey.

# Tag Info

Accepted

### How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
• 3,557
Accepted

### How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
• 9,326

• 266

### How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
• 1,773
Accepted

### Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
• 23.2k
Accepted

### How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
• 131

### What are the best hyper-parameters to tune in reinforcement learning?

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor ...
• 746

### Why is training longer not better in reinforcement learning?

Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
1 vote
Accepted

### PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that $$E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)$$ Here we abuse ...
• 93
1 vote

### Proximal Policy Optimization for continuous control problem

A different, variable reward structure might help. You could try a combination of airspeed, pitch, roll and whether it is hovering in the air or not in each timestep as a representation for the reward....
• 512
1 vote

### Proximal Policy Optimization for continuous control problem

Training using only 20 timesteps at a time is far too small, especially when the goal will ultimately consist of episodes of length 6000. You definitely need to increase that substantially and that ...
• 746
1 vote
Accepted

### How to define a continuous action distribution with a specific range for Reinforcement Learning?

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in \$...
• 4,026
1 vote

### Generation of 'new log probabilities' in continuous action space PPO

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...
• 522
1 vote

### Are these two TRPO objective functions equivalent?

As you point out, they are not equivalent. I guess you could store the time index for each state visited, but there are two problems with this. First, if you sample states according to their time ...
• 395
1 vote

### Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
1 vote

### If the average rewards start high and then decrease, could that indicate that the PPO is stuck at a local maximum?

I had the same problem where the reward kept decreasing and started to search for answers in the forum. I let the model trained while I search. As the model trained, the reward started to increase. ...
• 11
1 vote

### How is parallelism implemented in RL algorithms like PPO?

The paper Dota 2 with Large Scale Deep Reinforcement Learning goes into greater detail than the initial blog posts. They call their distributed training framework Rapid, which is also used in some ...
• 111
1 vote

### Why does the clipped surrogate objective work in Proximal Policy Optimization?

I think @16Aghnar explains the concept quite well. However, by clipping the surrogate objective alone doesn't ensure the trust region as stated in the paper: Engstrom et al., 2020, Implementation ...
• 226
1 vote

### How to implement a variable action space in Proximal Policy Optimization?

Normally, the set of actions that the agent can execute does not change over time, but some actions can become impossible in different states (for example, not every move is possible in any position ...

Only top scored, non community-wiki answers of a minimum length are eligible