8
votes
Accepted
How can policy gradients be applied in the case of multiple continuous actions?
As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
- 3,787
7
votes
Accepted
How to implement a variable action space in Proximal Policy Optimization?
The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
- 10.1k
7
votes
Accepted
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
- 491
6
votes
Why is the log probability replaced with the importance sampling in the loss function?
I am not 100% sure if the following is the only/complete story, but I'm quite confident it's at least part of the story:
In the PPO paper, after describing the standard policy gradient objective $L^{...
- 10.1k
5
votes
Accepted
What is the effect of parallel environments in reinforcement learning?
Do parallel environments improve the agent's ability to learn or does it not really make a difference?
Yes they can make a difference. There are two ways improvement is seen:
Collecting data from ...
- 30.3k
4
votes
Why is the log probability replaced with the importance sampling in the loss function?
For everybody getting here from google, like me: the $\log$ might have been replaced in the loss function, but I think it is still there when taking the gradient of both functions (correct me, if I am ...
- 81
4
votes
Accepted
Why don't we decorrelate transitions for policy-based data?
We do decorrelate training experience, even for policy gradient methods. This is because decorrelation helps training data be more like IID data, which helps with the convergence of SGD-like ...
- 2,038
4
votes
What are the pros and cons of using standard deviation or entropy for exploration in PPO?
Both implementations may be closer than you think.
In short:
PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
- 141
4
votes
What are the best hyper-parameters to tune in reinforcement learning?
You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL.
It suggests that discount factor ...
- 1,201
3
votes
What are the best hyper-parameters to tune in reinforcement learning?
Personally, I would choose the following two as the most important:
epsilon: When using an epsilon-greedy policy, epsilon determines how often the agent should explore and how often it should exploit....
- 532
3
votes
How is parallelism implemented in RL algorithms like PPO?
OpenAI have a post on that: https://openai.com/blog/openai-five/
They use a myriad of rollout workers that collect data for 60 seconds and push that data to a GPU cluster where gradients are computed ...
- 203
3
votes
Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs
I browsed through some other implementations of PPO and they all add small offset (1e-10) to prevent undefined log(0). I did that and the training works now.
- 191
3
votes
Accepted
Reinforcement Learning algorithm with rewards dependent both on previous action and current action
The answer to both your concerns is:
Add the previous action choice to the state representation.
It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
- 30.3k
3
votes
Accepted
How are continuous actions sampled (or generated) from the policy network in PPO?
As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :)
A common example for continuous spaces is the reparameterization trick, ...
- 151
3
votes
Accepted
Do we use validation and test sets for training a reinforcement learning agent?
No, we typically don't use a validation/test data set in Reinforcement Learning (RL). This is because of how we use the data in RL. The use of a data set is very different to the classic supervised/...
- 4,665
3
votes
Accepted
PPO: policy loss becomes nan
You might want to try substituting the exponentiation with a piecewise-defined function that uses a numerical approximation that is more numerically stable for low values of the exponent, such as ...
- 106
3
votes
Accepted
Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
Short answer:
The expectation $\mathbb{E}_t$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very ...
- 141
3
votes
Should DQN/PPO be used for state spaces that are not that large?
If I read correctly, your RL action space is a Multi-Discrete one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by ...
- 1,140
2
votes
Accepted
Why does the clipped surrogate objective work in Proximal Policy Optimization?
Ok, so I think I have a better understanding of this now.
Firstly, let's remind the main idea of the PPO : staying close to the previous policy. It's the same idea than in TRPO, but the L function is ...
- 601
2
votes
How to implement a variable action space in Proximal Policy Optimization?
Change the action space at each step, depending on the internal_state. I assume this is nonsense.
Yes, this seems overkill and makes the problem unnecessarily complex, there could be other things ...
2
votes
How do I calculate the policy in the Proximal Policy Optimization algorithm?
You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\...
- 266
2
votes
Generation of 'new log probabilities' in continuous action space PPO
The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...
- 552
2
votes
How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?
Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
- 2,426
2
votes
How does sharing parameters between the policy and value functions help in PPO?
Think of the network as a feature extractor followed by a policy head and a value function head. The feature extractor compresses the inputs into a lower dimensional feature vector that we hypothesize ...
- 1,126
2
votes
Accepted
PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$
Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that
$$
E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)
$$
Here we abuse ...
- 113
2
votes
Accepted
Can off-policy algorithms benefit from the parallelization?
From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
- 136
2
votes
How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?
You could take a look into options, (discrete-time) semi-MDPs, and multi-agent RL.
An option is a generalisation of an action. Mathematically, it's defined as a tuple $\langle\mathcal{I}, \pi, \beta\...
- 39.6k
2
votes
Why is training longer not better in reinforcement learning?
Could it be due to catastrophic forgetting/interference? If once the agent reaches 320K steps it becomes good at the task, it might start to experience only success. This could cause the agent to ...
- 21
2
votes
Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?
PPO is an algorithm in a class of actor-critic methods. In this class of methods, the training is broken in two counteracting parts: the "actor" parts learns the policy and the "critic&...
- 2,426
Only top scored, non community-wiki answers of a minimum length are eligible