Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
1 vote
0 answers
20 views

Why is training longer not better in reinforcement learning?

I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that. In the ...
user avatar
  • 203
0 votes
1 answer
37 views

How does PPO account for the last reward?

I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it ...
user avatar
1 vote
1 answer
36 views

How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?

I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
user avatar
  • 203
1 vote
0 answers
29 views

Normalisation of reward function

Problem Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
user avatar
  • 21
0 votes
0 answers
16 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
user avatar
  • 167
1 vote
0 answers
27 views

How to compare RL algorithms with different NN sizes?

I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC. When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
user avatar
  • 85
0 votes
1 answer
79 views

PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

So I've been looking at this formula for advantage estimate \begin{equation} \begin{aligned} & \hat{A}_t = \delta_t + (\gamma \lambda)\delta_{t+1} + ... + (\gamma \lambda)^{T-t+1}\delta_{T-1}\\ &...
user avatar
0 votes
2 answers
81 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
user avatar
  • 746
0 votes
0 answers
44 views

How to limit the action space and normalize at the same time in PPO?

In PPO (or TD3), how can you both determine the minimum and maximum action the agent can take (for example, between 0 and 1) and also make sure that all the actions sum up to 1? In Python, I can use <...
user avatar
1 vote
0 answers
30 views

How can PPO be combined with HER?

I ask because PPO is apparently an on-policy algorithm & the HER paper says that it can be combine with any off-policy algorithm. Yet I see GitHub projects that have combined them somehow? How is ...
user avatar
  • 266
0 votes
0 answers
27 views

Does it make sense to use the online network for making moves for both players in PPO for 2-player games?

I want to use PPO (or TRPO) to learn two-player games like 4 connect. Now, I'm thinking about whether or not it makes sense to use the online network for making moves for both players, in order to ...
user avatar
4 votes
1 answer
358 views

Do we use validation and test sets for training a reinforcement learning agent?

I am pretty new to reinforcement learning and was working with some code for the PPO and DQN algorithms. After looking at the code, I noticed that the authors did not include any code to setup a ...
user avatar
  • 187
1 vote
2 answers
72 views

Proximal Policy Optimization for continuous control problem

I am using clipped PPO to train a neural network to act as the controller for steering an aircraft, and am finding that my networks aren't learning. The goal is to keep the aircraft flying to cover ...
user avatar
0 votes
0 answers
85 views

Why does this PPO agent doesn't learn at all?

I am trying to code a Proximal Policy Optimization algorithm in Pytorch by myself for the OpenAI gym preduum-v0 environment. However, I find my learning curve is ...
user avatar
1 vote
1 answer
242 views

PPO when does the update happen?

In many places, it says PPO and Actor-Critic methods in general use TD-updates, but in the loss function for PPO, the Value function loss component uses the difference between output of the value ...
user avatar
  • 203
0 votes
1 answer
18 views

How to calculate policy probability ratio in multiple action space

I try to solve a navigation problem with PPO; my actions space have three-part: robot linear velocity that is in [-3,3] range (getting from a tanh activation func) robot linear angular that is in [-...
user avatar
  • 3
0 votes
0 answers
92 views

DDPG or PPO don't work well with my custom non gym environment

I have a project to control a robot with right and left wheel speeds, and my step time is not constant. Because my outputs are continuous (right wheel speed, left wheel speed, and time step), I try to ...
user avatar
  • 3
0 votes
0 answers
33 views

How to train critic network in PPO with multiple actor?

I try to code an algorithm to control a robot's movement with continuous action space. I use this question and create an actor network for each action.How can policy gradients be applied in the case ...
user avatar
  • 3
0 votes
0 answers
70 views

PPO: decreasing rewards as steps increase

How to explain this? Is this normal? Does this mean a bad design of the environment?
user avatar
  • 19
0 votes
0 answers
125 views

Playing Connect Four with reinforcement learning

I'm trying to do self-play reinforcement learning on a board game called Connect Four and I'm not getting good results so would appreciate some ideas on how to improve. I'm using the PPO algorithm ...
user avatar
  • 101
0 votes
1 answer
159 views

KL divergence coefficient update doesn't make sense in RLlib's PPO implementation

I am using RLlib (Ray 1.4.0)'s implementation of PPO for a multi-agent scenario with continuous actions, and I find that the loss includes the KL divergence penalty term, apart from the surrogate loss,...
user avatar
  • 203
0 votes
0 answers
21 views

PPO inputs giving different results in the same environment

I'm using Unity Ml-agents to learn more about reinforcement learning. I've created the most basic environment I could think of, but I'm getting some interesting results. Take a look at the picture ...
user avatar
0 votes
0 answers
55 views

tanh activation function output is not between -1 and 1 for continuous action PPO

I am using RLlib's (Ray = 1.4.0) PPO policy, and my first layer after the input (Conv layer) is producing a strange output keeping in mind that the activation for the output is a tanh, which I do ...
user avatar
  • 203
1 vote
1 answer
175 views

How to define a continuous action distribution with a specific range for Reinforcement Learning?

Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard ...
user avatar
  • 203
0 votes
0 answers
216 views

RLlib's Multi-agent PPO continuous actions turn into nan

After some amount of training on a custom Multi-agent sparse-reward environment using RLlib's (1.4.0) PPO network, I found that my continuous actions turn into nan (explodes?) which is probably caused ...
user avatar
  • 203
3 votes
2 answers
197 views

What are the best hyper-parameters to tune in reinforcement learning?

Obviously, this is somewhat subjective, but what hyper-parameters typically have the most significant impact on an RL agent's ability to learn? For example, the replay buffer size, learning rate, ...
user avatar
2 votes
1 answer
175 views

How does sharing parameters between the policy and value functions help in PPO?

The PPO objective may include a value function error term when parameters are shared between the policy and value functions. How does this help, and when to use a neural network architecture that ...
user avatar
  • 167
4 votes
1 answer
400 views

What is the effect of parallel environments in reinforcement learning?

Do parallel environments improve the agent's ability to learn or does it not really make a difference? Specifically, I am using PPO, but I think this applies across the board to other algorithms too.
user avatar
1 vote
1 answer
117 views

How should I interpret the surrogate and mean_noise_std plots of training a PPO model (from the Nvidia's Isaac gym)?

I am currently using the PPO method from the Nvidia's Isaac gym to train an agent for my robot. Below, you can see the plot which corresponds to a training process. I know that something is massively ...
user avatar
  • 21
1 vote
0 answers
350 views

PPO in continuous control not working

I have PPO agent for discrete action space for LunarLander-v2 env in gym and it works well. However, when i am trying to solve continuous version of the same env - <...
user avatar
1 vote
1 answer
125 views

Why is the logarithm of the standard deviation used in this implementation of proximal policy optimization?

I am currently writing my bachelor thesis, which is an implementation of proximal policy optimization. Sometimes, I hit a wall because of the gaps in my mathematical knowledge. However, implementing ...
user avatar
  • 13
1 vote
0 answers
25 views

Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL ...
user avatar
1 vote
0 answers
23 views

Understanding advantage estimator in proximal policy optimization

I was reading Proximal Policy Optimization paper. It states following: The advantage estimator used is: $\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T)...
user avatar
  • 211
1 vote
0 answers
54 views

Why is Openai's PPO2 implementation differentiable?

I'm trying to understand the concept behind the implementation of the OpenAI PPO2 algorithm. The loss function that is minimized is as follows: ...
user avatar
1 vote
0 answers
66 views

PPO2: Intuition behind Gumbel Softmax and Exploration?

I'm trying to understand the logic behind the magic of using the gumbel distribution for action sampling inside the PPO2 algorithm. This code snippet implements the action sampling, taken from here: <...
user avatar
2 votes
0 answers
236 views

Is (log-)standard deviation learned in TRPO and PPO or fixed instead?

After having read Williams (1992), where it was suggested that actually both the mean and standard deviation can be learned while training a REINFORCE algorithm on generating continuous output values, ...
user avatar
  • 707
2 votes
2 answers
328 views

Reinforcement Learning algorithm with rewards dependent both on previous action and current action

Problem description: Suppose we have an environment, where a reward at time step $t$ is dependent not only on the current action, but also on previous action in the following way: if current action ==...
user avatar
  • 23
1 vote
0 answers
140 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
user avatar
  • 167
1 vote
0 answers
300 views

What are the differences between Proximal Policy Optimization versions PPO1 and PPO2?

When Proximal Policy Optimization (PPO) was released, it was accompanied by a paper describing it. Later, the authors at OpenAI introduced a second version of PPO, called PPO2 (whereas the original ...
user avatar
  • 707
2 votes
1 answer
527 views

How are continuous actions sampled (or generated) from the policy network in PPO?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ ...
user avatar
  • 707
1 vote
0 answers
168 views

Why does PPO lead to a worse performance than TRPO in the same task?

I am training an agent with an Actor-Critic network and update it with TRPO so far. Now, I tried out PPO and the results are drastically different and bad. I only changed from TRPO to PPO, the rest of ...
user avatar
  • 31
1 vote
0 answers
73 views

How to design an observation(state) space for a simple `Rock-Paper-Scissor` game?

For weeks I've been working with this toy game of Rock-Paper-Scissor. I want to use a PPO agent learn to beat a computer ...
user avatar
  • 111
2 votes
1 answer
236 views

Generation of 'new log probabilities' in continuous action space PPO

I have a conceptual question for you all that hopefully I can convey clearly. I am building an RL agent in Keras using continuous PPO to control a laser attached to a pan/tilt turret for target ...
user avatar
  • 21
1 vote
0 answers
41 views

How can I increase the exploration in the Proximal Policy Optimation algorithm?

How can I increase the exploration in the Proximal Policy Optimation reinforcement learning algorithm? Is there a variable assigned for this purpose? I'm using the stable-baseline implementation: ...
user avatar
1 vote
1 answer
114 views

How to best make use of learning rate scheduling in reinforcement learning?

How to best make use of learning rate scheduling in reinforcement learning? To me, a low learning rate towards the end to fine-tune what you've learned with subtle updates makes sense. But I don't ...
user avatar
2 votes
0 answers
103 views

PPO algorithm converges on only one action

I have taken some reference implementations of PPO algorithm and am trying to create an agent which can play space invaders . Unfortunately from the 2nd trial onwards (after training the actor and ...
user avatar
2 votes
0 answers
104 views

Action masking for on policy algorithm like PPO

I have an environment, in which my agent learns according to PPO. The environment has a maximum of 80 actions, however not all of them are always allowed. My idea was to mask them, by setting the ...
user avatar
2 votes
0 answers
121 views

NaNs after a while in training of PPO

My problem is that every time I am trying to train my PPO agent I get NaN values after a while. The diagnostic that I get is the following: ...
user avatar
  • 151
3 votes
0 answers
90 views

How does normalization of the inputs work in the context of PPO?

What does the normalization of the inputs mean in the context of PPO? At each time step of an episode, I only know the values of this time step and of the previous ones, if I take track of them. This ...
user avatar
  • 151
3 votes
1 answer
200 views

What is the purpose of argmax in the PPO algorithm?

I'm kinda new to machine learning and still not too solid on math and particularly calculus. I'm currently trying to implement PPO algorithm as described in the spiningUp website : This line is ...
user avatar