8 votes
Accepted

How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
Jaden Travnik's user avatar
7 votes
Accepted

How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
Dennis Soemers's user avatar
  • 10.1k
7 votes
Accepted

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
Lee Reeves's user avatar
6 votes

Why is the log probability replaced with the importance sampling in the loss function?

I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story: In the PPO paper, after describing the standard policy gradient objective $L^{...
Dennis Soemers's user avatar
  • 10.1k
5 votes
Accepted

What is the effect of parallel environments in reinforcement learning?

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Yes they can make a difference. There are two ways improvement is seen: Collecting data from ...
Neil Slater's user avatar
  • 30.3k
4 votes

Why is the log probability replaced with the importance sampling in the loss function?

For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am ...
mglss's user avatar
  • 81
4 votes
Accepted

Why don't we decorrelate transitions for policy-based data?

We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like ...
Philip Raeisghasem's user avatar
4 votes

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

Both implementations may be closer than you think. In short: PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
FlorianH's user avatar
  • 141
4 votes

What are the best hyper-parameters to tune in reinforcement learning?

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor ...
Taw's user avatar
  • 1,201
3 votes

What are the best hyper-parameters to tune in reinforcement learning?

Personally, I would choose the following two as the most important: epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
devidduma's user avatar
  • 532
3 votes

How is parallelism implemented in RL algorithms like PPO?

OpenAI have a post on that: https://openai.com/blog/openai-five/ They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
Gregor's user avatar
  • 203
3 votes

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
Chintan Trivedi's user avatar
3 votes
Accepted

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
Neil Slater's user avatar
  • 30.3k
3 votes
Accepted

How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
kaiwenw's user avatar
  • 151
3 votes
Accepted

Do we use validation and test sets for training a reinforcement learning agent?

No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/...
David's user avatar
  • 4,665
3 votes
Accepted

PPO: policy loss becomes nan

You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
kmf's user avatar
  • 106
3 votes
Accepted

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

Short answer: The expectation $\mathbb{E}_t$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very ...
matorbi's user avatar
  • 141
3 votes

Should DQN/PPO be used for state spaces that are not that large?

If I read correctly, your RL action space is a Multi-Discrete one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by ...
Minh-Long Luu's user avatar
2 votes
Accepted

Why does the clipped surrogate objective work in Proximal Policy Optimization?

Ok, so I think I have a better understanding of this now. Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is ...
16Aghnar's user avatar
  • 601
2 votes

How to implement a variable action space in Proximal Policy Optimization?

Change the action space at each step, depending on the internal_state. I assume this is nonsense. Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things ...
Jithendaraa Subramanian's user avatar
2 votes

How do I calculate the policy in the Proximal Policy Optimization algorithm?

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\...
Omegastick's user avatar
2 votes

Generation of 'new log probabilities' in continuous action space PPO

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...
Hai Nguyen's user avatar
2 votes

How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
Kostya's user avatar
  • 2,426
2 votes

How does sharing parameters between the policy and value functions help in PPO?

Think of the network as a feature extractor followed by a policy head and a value function head. The feature extractor compresses the inputs into a lower dimensional feature vector that we hypothesize ...
harwiltz's user avatar
  • 1,126
2 votes
Accepted

PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that $$ E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t) $$ Here we abuse ...
vl_knd's user avatar
  • 113
2 votes
Accepted

Can off-policy algorithms benefit from the parallelization?

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
Scrimbibete's user avatar
2 votes

How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?

You could take a look into options, (discrete-time) semi-MDPs, and multi-agent RL. An option is a generalisation of an action. Mathematically, it's defined as a tuple $\langle\mathcal{I}, \pi, \beta\...
nbro's user avatar
  • 39.6k
2 votes

Why is training longer not better in reinforcement learning?

Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
Albert Rial's user avatar
2 votes

Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?

PPO is an algorithm in a class of actor-critic methods. In this class of methods, the training is broken in two counteracting parts: the "actor" parts learns the policy and the "critic&...
Kostya's user avatar
  • 2,426

Only top scored, non community-wiki answers of a minimum length are eligible