Questions tagged [proximal-policy-optimization]
For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.
109
questions
0
votes
0
answers
14
views
How to use LSTM ? RecurrentPPO from sb3-contrib
I'm working on creating an LSTM-based reinforcement learning model and trying to understand how Recurrent PPO from sb3-contrib works. Here's a simplified example of the code:
...
- 11
0
votes
1
answer
36
views
Understanding KL Stopping and KL Cutoff for the PPO algorithm
I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
- 588
0
votes
1
answer
74
views
Is my PPO agent learning? or is it just exploring?
I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src.
My doubts are mainly due ...
- 1
1
vote
1
answer
45
views
Filling replay buffer with expert trajectories for PPO/DQN
I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
- 41
1
vote
0
answers
69
views
Negative KL-divergence RLHF implementation
I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace:
What Is the Concern with Negative KL Divergence?
If you generate text by purely ...
- 11
1
vote
1
answer
61
views
How Can I Backpropagate My Network with PPO
I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There ...
- 25
0
votes
0
answers
20
views
Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm
I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
0
votes
1
answer
106
views
How do I add Entropy to a PPO algorithm?
I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
- 588
0
votes
2
answers
46
views
Reinforcement learning: Find the fastest solution (minimal number of steps)
I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines.
...
- 101
1
vote
1
answer
47
views
Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?
I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
- 41
0
votes
0
answers
34
views
Behaviour of PPO/similar Algos under action penalties
I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example:
Using PPO on a ...
0
votes
0
answers
139
views
Has anyone tried decision transformers with PPO?
I am trying to implement a PPO agent to try and solve (or at least get a good solution) for eternity 2 a tile matching game where each tile has 4 colored size you have to minimize the number of ...
1
vote
1
answer
66
views
Why do we not calculate value for the done state in PPO?
In PPO a common pattern I see in calcualting advantages is:
$$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$
Such as in this article. I am wondering why we multiply by <...
- 227
0
votes
0
answers
38
views
Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?
Domain: start at initial position, navigate to goal-position with N,S,E,W actions.
Algorithm: PPO
Libs: custom gym-env, stable baselines3
Penalties:
valid step: -1 (promote shortest path)
INvalid step:...
0
votes
0
answers
163
views
How does Proximal Policy Optimization deal with sparse reward
In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
- 175
0
votes
1
answer
58
views
How to expand an agent's action space with more actions?
I'm training a FPS agent using StableBaselines 3's PPO algorithm.
To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent ...
- 103
0
votes
1
answer
91
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 1
2
votes
1
answer
379
views
Should DQN/PPO be used for state spaces that are not that large?
I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
- 121
0
votes
0
answers
112
views
Why are the actor and critic losses look weird in my PPO implementation?
I tried implementing the PPO algorithm on the Mujoco environment (InvertedDoublePendulum - v2). During the training, the actor-loss started from 10^(-1) magnitude and converged at 10^(-3) magnitude. ...
1
vote
1
answer
184
views
PPO: dealing with variable episodic length
I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
2
votes
2
answers
114
views
Where does the proximal policy optimization objective's ratio term come from?
I will use the notation used in the proximal policy optimization paper.
What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?
Put ...
- 203
0
votes
0
answers
104
views
How to normalize a observation (state-vector) which is a mix of different types
I want to use an RL (DQN or PPO) for my use case. The use case is actually simple: The agent should change the behavior of a person. The person is busy with some tasks like reading a book or listening ...
0
votes
0
answers
89
views
Reinforcement learning PPO-clip agent returning softmax prediction of 1
I have discrete action space with 3 actions. I use distributed PPO-clip algorithm with these hyperparameters:
Workers: 32
Optimizer: Adam
Learning rate: 0.000005
Epochs: 20
Batch size: 256
Episode ...
- 1
3
votes
3
answers
217
views
Why clip the PPO objective on only one side?
In PPO with clipped surrogate objective (see the paper here), we have the following objective:
The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
- 31
1
vote
1
answer
156
views
Is there a reward function that would encourage exploration in this case?
I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is
...
0
votes
0
answers
182
views
PPO Agent - Unstable learning curve
I am using a PPO agent for a HVAC optimization problem. Trying to control a number of fans and heating coils in a network.
Currently, I am using 7 agents, each agent is responsible for one heating ...
0
votes
0
answers
72
views
PPO continous action space working in a complex scenario but failing to work in a simple scenario
I tried solving supply chain optimization problem using RL discrete and continuous actipn space. For some reason, with simplified version of problem (i.e. if customer order is always equal to 1), how ...
- 111
1
vote
1
answer
2k
views
Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?
I'm using PPO with an action-mask and I'm encountering a weird phenomenon.
At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
- 342
0
votes
1
answer
64
views
Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
- 41
0
votes
1
answer
56
views
Can TRPO use replay buffers?
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
Is it possible to ...
- 1
1
vote
0
answers
55
views
How many parameters of a RL environment?
I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters.
Given its stochasticity, I ...
1
vote
1
answer
51
views
How to use RL on a robotic moving arm?
I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
- 371
1
vote
1
answer
627
views
Why does a PPO agent perform only the action that costs the least?
I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator.
I would like to try to use parametric mode for actions, and ...
- 133
1
vote
0
answers
67
views
How can I get an integer as output for continuous action space PPO reinforcement learning?
I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
- 111
1
vote
0
answers
47
views
Can entropy bonus be used with state-independent log std for stochastic policies?
In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
- 131
3
votes
1
answer
187
views
Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
I understand how to derive the vanilla policy gradient
$$
\begin{align}
\nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
- 55
1
vote
1
answer
70
views
What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?
I was reading this link , and saw some creative architectures for PPO.
I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
- 342
3
votes
1
answer
4k
views
Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
- 53
1
vote
1
answer
480
views
Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?
I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
- 342
1
vote
2
answers
2k
views
How many training steps does it usually take to train an RL model?
This is my model average rewards as follow image.
How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?
And I'm using PPO to train.
- 283
1
vote
2
answers
835
views
Should PPO always converge toward the global optimum?
I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
- 13
1
vote
1
answer
360
views
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
- 53
1
vote
1
answer
699
views
PPO: multiple discrete actions per step, one depends on the other
I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.
Imagine ...
4
votes
1
answer
954
views
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
- 342
0
votes
0
answers
971
views
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
1
vote
1
answer
751
views
PPO: policy loss becomes nan [closed]
I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:
...
0
votes
1
answer
428
views
PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?
I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time.
In my current experiment, ...
2
votes
1
answer
547
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
- 213
0
votes
1
answer
197
views
How does PPO account for the last reward?
I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it ...
- 23
3
votes
2
answers
317
views
How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?
I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
- 223