Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
0
votes
0answers
23 views

How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
28 views

Patient PPO: how to handle imbalanced discrete action space?

PPO agent. The action space includes 3 actions: 0: do nothing 1: act (start) 2: stop The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
1
vote
0answers
30 views

How can I increase the exploration in the Proximal Policy Optimation algorithm?

How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: ...
1
vote
1answer
41 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
2
votes
0answers
30 views

PPO algorithm converges on only one action

I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and ...
2
votes
0answers
40 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
1
vote
0answers
28 views

NaNs after a while in training of PPO

My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following: ...
3
votes
0answers
31 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
3
votes
1answer
68 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
2
votes
0answers
36 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
1
vote
0answers
32 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
1
vote
0answers
22 views

Deciding std. deviation for policy network output?

When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
3
votes
1answer
109 views

Are these two TRPO objective functions equivalent?

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
2
votes
1answer
262 views

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations : Exploration with std Collect trajectories on $N$ timesteps, by using a policy-centered ...
2
votes
0answers
64 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
96 views

Understanding policy update in PPO2

I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation: From the original paper I know that the policy parameters $\theta$ are updated K-...
1
vote
1answer
372 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
2
votes
0answers
106 views

Understanding log probabilities of actions in the PPO objective

I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (code here), but I am confused about certain concepts. What is the correct way to implement log probability of a policy (...
1
vote
1answer
116 views

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I have implemented the total loss of my PPO objective as follows:- ...
2
votes
0answers
103 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
2
votes
0answers
280 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
0
votes
1answer
505 views

Reinforcement learning with PPO: rewards decreasing

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
4
votes
2answers
76 views

How is parallelism implemented in RL algorithms like PPO?

There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory ...
0
votes
1answer
42 views

Why don't we decorrelate transitions for policy-based data?

I'm implementing PPO myself strictly follow the steps: sample transitions randomly shuffle the sampled transitions compute gradients and update networks using the sampled transitions drop transitions ...
0
votes
1answer
548 views

Getting NaN from A3C PPO model [closed]

I've pieced together this A3C w/ PPO Gym Pendulum example, but I'm finding after a while, when attempting to get the action from the model, I get a NaN return: ...
2
votes
1answer
475 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
6
votes
2answers
680 views

Why is the log probability replaced with the importance sampling in the loss function?

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients $$...
11
votes
1answer
6k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...