Questions tagged [proximal-policy-optimization]
For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.
102
questions
0
votes
0
answers
27
views
Behaviour of PPO/similar Algos under action penalties
I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example:
Using PPO on a ...
0
votes
0
answers
57
views
Has anyone tried decision transformers with PPO?
I am trying to implement a PPO agent to try and solve (or at least get a good solution) for eternity 2 a tile matching game where each tile has 4 colored size you have to minimize the number of ...
1
vote
1
answer
38
views
Why do we not calculate value for the done state in PPO?
In PPO a common pattern I see in calcualting advantages is:
$$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$
Such as in this article. I am wondering why we multiply by <...
- 227
0
votes
0
answers
28
views
Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?
Domain: start at initial position, navigate to goal-position with N,S,E,W actions.
Algorithm: PPO
Libs: custom gym-env, stable baselines3
Penalties:
valid step: -1 (promote shortest path)
INvalid step:...
0
votes
0
answers
58
views
How does Proximal Policy Optimization deal with sparse reward
In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
- 173
0
votes
1
answer
45
views
How to expand an agent's action space with more actions?
I'm training a FPS agent using StableBaselines 3's PPO algorithm.
To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent ...
- 103
0
votes
1
answer
62
views
What should I do, reinforcement learning agent gives different result on every train?
I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
- 1
2
votes
1
answer
199
views
Should DQN/PPO be used for state spaces that are not that large?
I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
- 121
0
votes
0
answers
76
views
Why are the actor and critic losses look weird in my PPO implementation?
I tried implementing the PPO algorithm on the Mujoco environment (InvertedDoublePendulum - v2). During the training, the actor-loss started from 10^(-1) magnitude and converged at 10^(-3) magnitude. ...
1
vote
1
answer
128
views
PPO: dealing with variable episodic length
I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
2
votes
1
answer
71
views
Where does the proximal policy optimization objective's ratio term come from?
I will use the notation used in the proximal policy optimization paper.
What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?
Put ...
- 203
0
votes
0
answers
57
views
How to normalize a observation (state-vector) which is a mix of different types
I want to use an RL (DQN or PPO) for my use case. The use case is actually simple: The agent should change the behavior of a person. The person is busy with some tasks like reading a book or listening ...
0
votes
0
answers
78
views
Reinforcement learning PPO-clip agent returning softmax prediction of 1
I have discrete action space with 3 actions. I use distributed PPO-clip algorithm with these hyperparameters:
Workers: 32
Optimizer: Adam
Learning rate: 0.000005
Epochs: 20
Batch size: 256
Episode ...
- 1
3
votes
3
answers
169
views
Why clip the PPO objective on only one side?
In PPO with clipped surrogate objective (see the paper here), we have the following objective:
The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
- 31
0
votes
1
answer
129
views
Is there a reward function that would encourage exploration in this case?
I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is
...
0
votes
0
answers
125
views
PPO Agent - Unstable learning curve
I am using a PPO agent for a HVAC optimization problem. Trying to control a number of fans and heating coils in a network.
Currently, I am using 7 agents, each agent is responsible for one heating ...
0
votes
0
answers
64
views
PPO continous action space working in a complex scenario but failing to work in a simple scenario
I tried solving supply chain optimization problem using RL discrete and continuous actipn space. For some reason, with simplified version of problem (i.e. if customer order is always equal to 1), how ...
- 111
1
vote
1
answer
1k
views
Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?
I'm using PPO with an action-mask and I'm encountering a weird phenomenon.
At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
- 322
0
votes
1
answer
60
views
Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?
My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
- 41
0
votes
0
answers
36
views
Can TRPO use replay buffers?
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
Is it possible to ...
- 1
1
vote
0
answers
52
views
How many parameters of a RL environment?
I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters.
Given its stochasticity, I ...
1
vote
1
answer
35
views
How to use RL on a robotic moving arm?
I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
- 361
1
vote
1
answer
467
views
Why does a PPO agent perform only the action that costs the least?
I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator.
I would like to try to use parametric mode for actions, and ...
- 133
1
vote
0
answers
57
views
How can I get an integer as output for continuous action space PPO reinforcement learning?
I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
- 111
1
vote
0
answers
40
views
Can entropy bonus be used with state-independent log std for stochastic policies?
In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
- 131
0
votes
0
answers
71
views
Model learning only to not to trade in a static sequence
I am learning RL and using out-of-the-box Stable-Baselines 3 PPO algorithm, I made a custom environment with the following ...
- 447
3
votes
1
answer
160
views
Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
I understand how to derive the vanilla policy gradient
$$
\begin{align}
\nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
- 55
1
vote
0
answers
40
views
What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?
I was reading this link , and saw some creative architectures for PPO.
I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
- 322
1
vote
1
answer
3k
views
Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?
I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
- 33
1
vote
1
answer
385
views
Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?
I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
- 322
0
votes
0
answers
82
views
Is the interpretation of the "batch size" in policy gradient algorithms the number of trajectories sampled in VPG and TRPO?
I would like to shore up my interpretation of the concept of "batch size".
It is my understanding that in Vanilla Policy Gradients and TRPO, the "batch size" is the number of ...
0
votes
0
answers
33
views
Random Network Distillation for short episodic tasks
I have a task with short episodic steps (maybe up to 20) that I'm training for using PPO, but it seems to get stuck easily on local optima. Searching for solutions to this, I've stumbled upon Random ...
1
vote
2
answers
1k
views
How many training steps does it usually take to train an RL model?
This is my model average rewards as follow image.
How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model?
And I'm using PPO to train.
- 283
1
vote
2
answers
627
views
Should PPO always converge toward the global optimum?
I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
- 13
1
vote
1
answer
253
views
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
- 33
1
vote
1
answer
541
views
PPO: multiple discrete actions per step, one depends on the other
I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.
Imagine ...
4
votes
1
answer
947
views
Mathematically, what is happening differently in the neural net during exploration vs. exploitation?
I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
- 322
0
votes
0
answers
775
views
PPO: how to scale rewards
I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
1
vote
1
answer
639
views
PPO: policy loss becomes nan [closed]
I'm implement PPO for a very specific problem, and it seems to be working somewhat, but after a few epochs, I always get something like this:
...
0
votes
1
answer
313
views
PPO custom implementation: do metrics like value loss, actor loss and entropy move a certain way?
I'm wondering whether problems with a custom PPO implementation (problem couldn't be made into a gym environment) can be debugged considering how the losses change over time.
In my current experiment, ...
2
votes
1
answer
474
views
Why is training longer not better in reinforcement learning?
I have trained an RL agent (PPO) for 6 million steps to solve the OpenAI gym LunarLander-v2. Surprisingly, the agent performs best already after 320K steps and is getting worse after that.
In the ...
- 213
0
votes
1
answer
150
views
How does PPO account for the last reward?
I was implementing PPO for the lunar lander environment in openai gym, but my agent seems to be getting stuck at a score of ~-80. On the website it says the agent gets rewarded +100 or -100 if it ...
- 23
3
votes
2
answers
271
views
How to model a multi-agent reinforcement learning problem where actions of different agents can take different durations?
I am confused on a conceptual scale how I would be able to model a multi-agent reinforcement learning problem when each agent performing an action would take different durations to complete the action....
- 223
1
vote
0
answers
64
views
Normalisation of reward function
Problem
Currently, I have some problems defining a reward function for my RL project and mainly with how to normalise the score such that the highest possible score for all instances of the ...
- 31
2
votes
1
answer
178
views
Can off-policy algorithms benefit from the parallelization?
On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
- 331
1
vote
0
answers
47
views
How to compare RL algorithms with different NN sizes?
I wanted to run some tests with some RL algorithms in a continuous control task, namely PPO-clip and SAC.
When comparing their NN structures described in their papers, SAC used 2 layers with 256 ...
- 115
0
votes
1
answer
477
views
PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$
So I've been looking at this formula for advantage estimate
\begin{equation}
\begin{aligned}
& \hat{A}_t = \delta_t + (\gamma \lambda)\delta_{t+1} + ... + (\gamma \lambda)^{T-t+1}\delta_{T-1}\\
&...
- 697
0
votes
2
answers
284
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 1,131
2
votes
0
answers
247
views
How can PPO be combined with HER?
I ask because PPO is apparently an on-policy algorithm & the HER paper says that it can be combine with any off-policy algorithm. Yet I see GitHub projects that have combined them somehow?
How is ...
- 326
4
votes
1
answer
2k
views
Do we use validation and test sets for training a reinforcement learning agent?
I am pretty new to reinforcement learning and was working with some code for the PPO and DQN algorithms. After looking at the code, I noticed that the authors did not include any code to setup a ...
- 197