# Tag Info

Accepted

### How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
• 3,667
Accepted

### How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
• 9,649
Accepted

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...
• 461

• 266

### How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...
• 1,972
Accepted

### Reinforcement Learning algorithm with rewards dependent both on previous action and current action

The answer to both your concerns is: Add the previous action choice to the state representation. It is all you need to do. It gives the agent the data it needs to learn the association of negative ...
• 25.4k
Accepted

### How are continuous actions sampled (or generated) from the policy network in PPO?

As long as your policy (propensity) is differentiable, everything's is good. Discrete, continuous, other, doesn't matter! :) A common example for continuous spaces is the reparameterization trick, ...
• 141

### What are the best hyper-parameters to tune in reinforcement learning?

You should read this study https://arxiv.org/abs/2006.05990 which does some empirical study on this question, specifically for on-policy, continuous action space DRL. It suggests that discount factor ...
• 1,111

### How does sharing parameters between the policy and value functions help in PPO?

Think of the network as a feature extractor followed by a policy head and a value function head. The feature extractor compresses the inputs into a lower dimensional feature vector that we hypothesize ...
• 1,061
Accepted

### Can off-policy algorithms benefit from the parallelization?

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
• 136