Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
2 votes
1 answer
52 views

How does reward work while training a Reinforcement Learning agent?

I am using PPO to train my environment which I created using stable baselines 3. I am confused if I should make the reward = 0 in the step function or not. Initially, I used to have self.reward = 0 in ...
user79474's user avatar
0 votes
0 answers
37 views

PPO agent of path planning does not learn

I built an agent based on PPO with actor-critic structure for a project of path planning, in which the objective of agent is to find a path toward the goal bypass the obstacle in a 16*16 grid map. ...
speedhawk1's user avatar
0 votes
0 answers
26 views

What is the Input to the agent in the paper "Model Based Reinforcement Learning for Atari", and why does the world model run at inference time?

I am currently reading the paper Model based Reinforcement Learning for Atari. However, they do not specify what exactly they use as input for the agent. I believe it to be the observation space - ...
Nick's user avatar
  • 1
2 votes
0 answers
32 views

How does PPO with advantage normalization learn in MountainCar-v0 before first reaching the goal state?

I'm trying to figure out how PPO ever learns anything in a sparse environment like gymnasium's MountainCar-v0 before it first ever reaches the goal state. Specifically was looking at stable_baselines3'...
Switch's user avatar
  • 121
0 votes
0 answers
20 views

Does PPO clip kills gradient

In PPO Loss function, the reasoning to use the clip function is to do the update within a trusted region. But once we use the clip function, don't it kills the gradients with it? Because the only ...
Wenuka's user avatar
  • 101
0 votes
0 answers
30 views

Reinforcement Learning agent converging at the lowest reward

I have a custom environment created using Stable Baselines 3 where the environment is a digital twin of a fermentation reaction. It observes the enzyme activity which is the output of the fermentation ...
user79474's user avatar
1 vote
0 answers
15 views

Enhancing Generalization in DRL Agents in Static Data Environments

Context: I'm working with a deep reinforcement learning (DRL) agent in a market-like environment where its actions do not affect the environment. The environment uses historical data up to a certain ...
ElonMuskofBadIdeas's user avatar
0 votes
1 answer
162 views

Is PPO a policy-based method or an actor-critique-based method?

as far as i understand there are 3 categories of Reinforcement algorithms: Value-based methods (like DQN or Sarsa) Policy-based methods (like REINFORCE) Actor-critic-based methods (like A2C) To ...
PeterBe's user avatar
  • 246
0 votes
1 answer
51 views

In the PPO objective function, does the expectation do a weighted average over the batch? [closed]

I am currently working on implementing Proximal Policy Optimization (PPO) for a reinforcement learning task, and I have a few questions regarding the objective function and probability calculations. ...
TWTom's user avatar
  • 3
0 votes
0 answers
44 views

What algorithm should I use to train an antichess agent?

I will implement an antichess agent and am not sure about which algorithm to use. My current candidates are minimax with alpha-beta pruning, MCTS and proximal policy optimization. Should I consider ...
heyula's user avatar
  • 1
2 votes
0 answers
64 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
0 votes
1 answer
118 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
0 votes
1 answer
171 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
2 votes
1 answer
146 views

Filling replay buffer with expert trajectories for PPO/DQN

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
Erik Storm's user avatar
1 vote
0 answers
155 views

Negative KL-divergence RLHF implementation

I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace: What Is the Concern with Negative KL Divergence? If you generate text by purely ...
probably45's user avatar
1 vote
1 answer
128 views

How Can I Backpropagate My Network with PPO

I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There ...
Ege's user avatar
  • 25
0 votes
0 answers
33 views

Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm

I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
Mohammad Hosseini's user avatar
0 votes
1 answer
366 views

How do I add Entropy to a PPO algorithm?

I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
desert_ranger's user avatar
0 votes
2 answers
106 views

Reinforcement learning: Find the fastest solution (minimal number of steps)

I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines. ...
P. Egli's user avatar
  • 101
2 votes
1 answer
100 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
0 votes
0 answers
38 views

Behaviour of PPO/similar Algos under action penalties

I am currently experimenting with PPO in different environments. I am interested in learning policies that fulfill a certain goal while keeping a specific value low. Here's an example: Using PPO on a ...
Lukas Schroth's user avatar
0 votes
0 answers
268 views

Has anyone tried decision transformers with PPO?

I am trying to implement a PPO agent to try and solve (or at least get a good solution) for eternity 2 a tile matching game where each tile has 4 colored size you have to minimize the number of ...
mt-clemente's user avatar
1 vote
1 answer
110 views

Why do we not calculate value for the done state in PPO?

In PPO a common pattern I see in calcualting advantages is: $$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$ Such as in this article. I am wondering why we multiply by <...
Jacob B's user avatar
  • 247
0 votes
0 answers
57 views

Penalizing invalid action and ending episode early causes agent to end episode with invalid action in a non-goal state?

Domain: start at initial position, navigate to goal-position with N,S,E,W actions. Algorithm: PPO Libs: custom gym-env, stable baselines3 Penalties: valid step: -1 (promote shortest path) INvalid step:...
marshacker's user avatar
0 votes
0 answers
362 views

How does Proximal Policy Optimization deal with sparse reward

In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
Sam's user avatar
  • 185
0 votes
1 answer
61 views

How to expand an agent's action space with more actions?

I'm training a FPS agent using StableBaselines 3's PPO algorithm. To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent ...
Ilija Vuk's user avatar
  • 103
1 vote
1 answer
136 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
ad124j2's user avatar
  • 11
2 votes
1 answer
583 views

Should DQN/PPO be used for state spaces that are not that large?

I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
gameveloster's user avatar
1 vote
1 answer
242 views

PPO: dealing with variable episodic length

I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
Antonis Karvelas's user avatar
2 votes
2 answers
175 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
fool's user avatar
  • 203
3 votes
4 answers
318 views

Why clip the PPO objective on only one side?

In PPO with clipped surrogate objective (see the paper here), we have the following objective: The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
Jer's user avatar
  • 31
1 vote
1 answer
186 views

Is there a reward function that would encourage exploration in this case?

I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is ...
Sai Dinesh Pola's user avatar
1 vote
1 answer
2k views

Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?

I'm using PPO with an action-mask and I'm encountering a weird phenomenon. At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
Vladimir Belik's user avatar
0 votes
1 answer
67 views

Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
Magi Feeney's user avatar
0 votes
1 answer
91 views

Can TRPO use replay buffers?

I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$. Is it possible to ...
Wj210's user avatar
  • 1
1 vote
0 answers
66 views

How many parameters of a RL environment?

I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters. Given its stochasticity, I ...
Damuna Taliffato's user avatar
1 vote
1 answer
80 views

How to use RL on a robotic moving arm?

I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
Hadar Sharvit's user avatar
1 vote
1 answer
743 views

Why does a PPO agent perform only the action that costs the least?

I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator. I would like to try to use parametric mode for actions, and ...
Francesco's user avatar
  • 133
1 vote
0 answers
83 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
NAnn's user avatar
  • 111
1 vote
0 answers
54 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
3 votes
1 answer
214 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
Peter's user avatar
  • 55
1 vote
1 answer
121 views

What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?

I was reading this link , and saw some creative architectures for PPO. I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
Vladimir Belik's user avatar
3 votes
1 answer
6k views

Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
Aeryan's user avatar
  • 53
1 vote
1 answer
598 views

Why does mean episode reward during training differ dramatically from "manual" runs of the trained model on same data?

I am training an RL agent, using PPO, on a time-series environment that comes from a tabular dataset. The possible scores during an episode goes from -1 to positive infinity (though realistically, I ...
Vladimir Belik's user avatar
3 votes
2 answers
2k views

How many training steps does it usually take to train an RL model?

This is my model average rewards as follow image. How to tell if it is undertrained or not convergent? How many training steps does it usually take to train an RL model? And I'm using PPO to train.
huang's user avatar
  • 305
1 vote
2 answers
1k views

Should PPO always converge toward the global optimum?

I'm trying to "solve" the OpenAI gym environment "Humanoid-v3" using PPO. I got it to work to some degree (The NN is learning a policy and perfecting it. Average reward of about 5....
pjungk's user avatar
  • 13
1 vote
1 answer
498 views

How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
Aeryan's user avatar
  • 53
1 vote
1 answer
875 views

PPO: multiple discrete actions per step, one depends on the other

I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other. Imagine ...
Antonis Karvelas's user avatar
4 votes
1 answer
956 views

Mathematically, what is happening differently in the neural net during exploration vs. exploitation?

I want to understand roughly what is happening in the neural network of an RL agent when it is exploring vs. exploiting. For example, are the network weights not being updated when the agent is ...
Vladimir Belik's user avatar
0 votes
0 answers
1k views

PPO: how to scale rewards

I have a custom PPO implementation and a problem that has costs rather than rewards, so I basically need to take the negative value for PPO to work. As the values are somewhat large, I've tried ...
Antonis Karvelas's user avatar