# Tag Info

Accepted

### How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
Accepted

### How to implement a variable action space in Proximal Policy Optimization?

The most straightforward solution is to simply make every action "legal", but implementing a consistent, deterministic mapping from potentially illegal actions to different legal actions. Whenever the ...
Accepted

### Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

Typically, the NN is trained the same way whether an action is chosen for exploration or exploitation. Look at the objective (AKA loss) function for any algorithm you're interested in and you'll ...

### Generation of 'new log probabilities' in continuous action space PPO

The idea in PPO is that you want to reuse the batch many times to update the current policy. However, you cannot update mindlessly in a regular actor-critic fashion, because your policy might stray ...

### How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

Loss. In the context of Deep Learning and Deep Reinforcement Learning, "training" is just a fancy word for "optimization". You are essentially looking for an optimum point in some ...

### How does sharing parameters between the policy and value functions help in PPO?

Think of the network as a feature extractor followed by a policy head and a value function head. The feature extractor compresses the inputs into a lower dimensional feature vector that we hypothesize ...
Accepted

### PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that $$E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)$$ Here we abuse ...
Accepted

### Can off-policy algorithms benefit from the parallelization?

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...

### How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?

You could take a look into options, (discrete-time) semi-MDPs, and multi-agent RL. An option is a generalisation of an action. Mathematically, it's defined as a tuple \$\langle\mathcal{I}, \pi, \beta\...