As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

102 questions
Filter by
Sorted by
Tagged with
27 views

Behaviour of PPO/similar Algos under action penalties

I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example: Using PPO on a ...
57 views

Has anyone tried decision transformers with PPO?

I am trying to implement a PPO agent to try and solve (or at least get a good solution) for eternity 2 a tile matching game where each tile has 4 colored size you have to minimize the number of ...
1 vote
38 views

Why do we not calculate value for the done state in PPO?

In PPO a common pattern I see in calcualting advantages is: $$delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$ Such as in this article. I am wondering why we multiply by <...
• 227
28 views

Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?

Domain: start at initial position, navigate to goal-position with N,S,E,W actions. Algorithm: PPO Libs: custom gym-env, stable baselines3 Penalties: valid step: -1 (promote shortest path) INvalid step:...
58 views

How does Proximal Policy Optimization deal with sparse reward

In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
• 173
45 views

How to expand an agent's action space with more actions?

I'm training a FPS agent using StableBaselines 3's PPO algorithm. To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent ...
• 103
62 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
199 views

Should DQN/PPO be used for state spaces that are not that large?

I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
• 121
76 views

Why are the actor and critic losses look weird in my PPO implementation?

I tried implementing the PPO algorithm on the Mujoco environment (InvertedDoublePendulum - v2). During the training, the actor-loss started from 10^(-1) magnitude and converged at 10^(-3) magnitude. ...
1 vote
128 views

PPO: dealing with variable episodic length

I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
71 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
• 203
57 views

How to normalize a observation (state-vector) which is a mix of different types

I want to use an RL (DQN or PPO) for my use case. The use case is actually simple: The agent should change the behavior of a person. The person is busy with some tasks like reading a book or listening ...
78 views

Reinforcement learning PPO-clip agent returning softmax prediction of 1

I have discrete action space with 3 actions. I use distributed PPO-clip algorithm with these hyperparameters: Workers: 32 Optimizer: Adam Learning rate: 0.000005 Epochs: 20 Batch size: 256 Episode ...
• 1
169 views

Why clip the PPO objective on only one side?

In PPO with clipped surrogate objective (see the paper here), we have the following objective: The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
• 31
129 views

Is there a reward function that would encourage exploration in this case?

I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is ...
125 views

PPO Agent - Unstable learning curve

I am using a PPO agent for a HVAC optimization problem. Trying to control a number of fans and heating coils in a network. Currently, I am using 7 agents, each agent is responsible for one heating ...
64 views

PPO continous action space working in a complex scenario but failing to work in a simple scenario

I tried solving supply chain optimization problem using RL discrete and continuous actipn space. For some reason, with simplified version of problem (i.e. if customer order is always equal to 1), how ...
• 111
1 vote
1k views

Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?

I'm using PPO with an action-mask and I'm encountering a weird phenomenon. At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
60 views

Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
36 views

Can TRPO use replay buffers?

I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$. Is it possible to ...
1 vote
52 views

How many parameters of a RL environment?

I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters. Given its stochasticity, I ...
1 vote
35 views

How to use RL on a robotic moving arm?

I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
1 vote
467 views

Why does a PPO agent perform only the action that costs the least?

I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator. I would like to try to use parametric mode for actions, and ...
• 133
1 vote
57 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
• 111
1 vote
40 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
• 131
71 views

Model learning only to not to trade in a static sequence

I am learning RL and using out-of-the-box Stable-Baselines 3 PPO algorithm, I made a custom environment with the following ...
• 447
160 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient  \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
• 55
1 vote
40 views

What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?

I was reading this link , and saw some creative architectures for PPO. I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
1 vote
3k views

Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
• 33
1 vote
385 views

Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?

I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
82 views

Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?

I would like to shore up my interpretation of the concept of "batch size". It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the number of ...
33 views

Random Network Distillation for short episodic tasks

I have a task with short episodic steps (maybe up to 20) that I'm training for using PPO, but it seems to get stuck easily on local optima. Searching for solutions to this, I've stumbled upon Random ...
1 vote
1k views

How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
• 283
1 vote
627 views

Should PPO always converge toward the global optimum?

I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
• 13
1 vote
253 views

How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
• 33
1 vote
541 views

PPO: multiple discrete actions per step, one depends on the other

I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other. Imagine ...
947 views

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
775 views

PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
1 vote
639 views

PPO: policy loss becomes nan [closed]

I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this: ...
313 views

PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?

I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time. In my current experiment, ...
474 views

Why is training longer not better in reinforcement learning?

I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that. In the ...
• 213
150 views

How does PPO account for the last reward?

I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it ...
271 views

How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?

I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
• 223
1 vote
64 views

Normalisation of reward function

Problem Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
• 31
178 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
• 331
1 vote
47 views

How to compare RL algorithms with different NN sizes?

I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC. When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
• 115
477 views

PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

So I've been looking at this formula for advantage estimate \begin{aligned} & \hat{A}_t = \delta_t + (\gamma \lambda)\delta_{t+1} + ... + (\gamma \lambda)^{T-t+1}\delta_{T-1}\\ &...
• 697
284 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
• 1,131
247 views

How can PPO be combined with HER?

I ask because PPO is apparently an on-policy algorithm & the HER paper says that it can be combine with any off-policy algorithm. Yet I see GitHub projects that have combined them somehow? How is ...
• 326