Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
1
vote
2answers
32 views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem desciption: Suppose we have an environment, where a reward at time step t is dependent not only on the current action, but also on previous action in the following way: if $current$ $action$ $...
1
vote
0answers
34 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
1
vote
0answers
28 views

What are the differences between Proximal Policy Optimization versions PPO1 and PPO2?

When Proximal Policy Optimization (PPO) was released, it was accompanied by a paper describing it. Later, the authors at OpenAI introduced a second version of PPO, called PPO2 (whereas the original ...
1
vote
1answer
59 views

How are continuous actions sampled (or generated) from the policy network in PPO?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ ...
1
vote
0answers
43 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
1
vote
0answers
42 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
3
votes
1answer
43 views

Generation of 'new log probabilities' in continuous action space PPO

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target ...
1
vote
0answers
25 views

How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
31 views

Patient PPO: how to handle imbalanced discrete action space?

PPO agent. The action space includes 3 actions: 0: do nothing 1: act (start) 2: stop The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
1
vote
0answers
34 views

How can I increase the exploration in the Proximal Policy Optimation algorithm?

How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: ...
2
votes
1answer
57 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
2
votes
0answers
34 views

PPO algorithm converges on only one action

I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and ...
2
votes
0answers
48 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
1
vote
0answers
43 views

NaNs after a while in training of PPO

My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following: ...
3
votes
0answers
33 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
3
votes
1answer
97 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
2
votes
0answers
41 views

Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
2
votes
0answers
37 views

Should I consider mean or sampled value for action selection in ppo algorithm?

When considering the policy network in PPO algorithm, we need to fit a Gaussian distribution to the neural network output (for a continuous action space problem). When I use this network to obtain ...
1
vote
0answers
23 views

Deciding std. deviation for policy network output?

When I try to fit a Normal Distribution to the output of a policy network, for a continuous action space problem, what should be its standard deviation? mean for the distribution will directly be the ...
4
votes
1answer
121 views

Are these two TRPO objective functions equivalent?

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
3
votes
1answer
428 views

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

When trying to implement my own PPO (Proximal Policy Optimizer), I came across two different implementations : Exploration with std Collect trajectories on $N$ timesteps, by using a policy-centered ...
2
votes
0answers
73 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
1
vote
0answers
115 views

Understanding policy update in PPO2

I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation: From the original paper I know that the policy parameters $\theta$ are updated K-...
1
vote
1answer
551 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
3
votes
0answers
150 views

Understanding log probabilities of actions in the PPO objective

I'm trying to implement the Proximal Policy Optimization (PPO) algorithm (code here), but I am confused about certain concepts. What is the correct way to implement log probability of a policy (...
1
vote
1answer
182 views

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I have implemented the total loss of my PPO objective as follows:- ...
2
votes
0answers
108 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
2
votes
0answers
358 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
0
votes
1answer
711 views

Reinforcement learning with PPO: rewards decreasing

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
4
votes
2answers
147 views

How is parallelism implemented in RL algorithms like PPO?

There are multiple ways to implement parallelism in reinforcement learning. One is to use parallel workers running in their own environments to collect data in parallel, instead of using replay memory ...
0
votes
1answer
53 views

Why don't we decorrelate transitions for policy-based data?

I'm implementing PPO myself strictly follow the steps: sample transitions randomly shuffle the sampled transitions compute gradients and update networks using the sampled transitions drop transitions ...
0
votes
1answer
687 views

Getting NaN from A3C PPO model [closed]

I've pieced together this A3C w/ PPO Gym Pendulum example, but I'm finding after a while, when attempting to get the action from the model, I get a NaN return: ...
2
votes
1answer
564 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
13
votes
3answers
4k views

How to implement a variable action space in Proximal Policy Optimization?

I'm coding a Proximal Policy Optimization (PPO) agent with the Tensorforce library (which is built on top of TensorFlow). The first environment was very simple. Now, I'm diving into a more complex ...
6
votes
2answers
778 views

Why is the log probability replaced with the importance sampling in the loss function?

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients $$L^...
4
votes
1answer
231 views

Understanding multi-iteration updates of the model in the Proximal Policy Optimization algorithm

I have a general question about the updating of the network/model in the PPO algorithm. If I understand it correctly, there are multiple iterations of weight updates done on the model with data that ...
13
votes
1answer
6k views

How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms. When using a single continuous action, normally, you would use some ...