Skip to main content

Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
-1 votes
0 answers
21 views

RL PPO Algorithem Doesn't Learn with Fixed Sigma?

I have a problem with my PPO Implementation in Python with Jax. My Agent for example optimizes the ant-v4 environment but only if the sigmas out of my Neural Net of the Policy are be between [1e-6,1]. ...
PC Benutzer's user avatar
0 votes
1 answer
31 views

Can all RL algorithms learn with discrete state spaces?

This question come to mind when i was planing to do a benchmark of RL algorithms to my Environment. In fact, Q-Learning, SARSA actually only handles with discrete state spaces because they are tabular ...
Vitor Martins's user avatar
0 votes
0 answers
22 views

Why do we need the min in PPO objective function?

I don't understand why we need to take the minimum of the unclipped part and the clipped part in PPO's objective function. Why not just use the clipped term? How can the clipped part ever be bigger ...
mavex857's user avatar
0 votes
0 answers
48 views

What's the action space in RLHF for LLM?

I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
Druudik's user avatar
  • 171
0 votes
0 answers
25 views

Difficulty training PPO agent for robotic arm navigation task

 I'm currently working on training a PPO agent for a robotic arm navigation task, where the goal is to navigate the robotic arm to different positions in the environment. I've been training the agent ...
Weitao Kang's user avatar
0 votes
0 answers
12 views

How to clip actions based on state in RL environment for trading (and other tips to approach optimal trade execution)?

I’m currently trying to use RL to analyze price impact in financial markets for optimal trade execution and have coded up a custom gymnasium environment to do so. Now, I'm deciding on which RL ...
walrus's user avatar
  • 1
0 votes
0 answers
43 views

why PPO performs better than TRPO?

what is the logistic behind why PPO performs better than TRPO(at least in the PPO's paper results)? When I read both papers, it seems like PPO is just simpler version of TRPO, that it does not run ...
user3315463's user avatar
0 votes
1 answer
20 views

InstructGPT: In the objective RL function, does $y$ mean full response, or only a single token?

In the paper, They write: Now, is $y$ the full response or only the next token repsonse? One the one hand, the reward model expects full response, on the other hand they write 'per-token KL penalty'. ...
Nathan G's user avatar
  • 161
1 vote
0 answers
81 views

Reward shaping for dense and sparse rewards

I am working on an RL Problem that drives me nuts. My goal is to control a robot arm in a simulator that has to do 2 things: Hold the arm in a certain position (that is easy and done) If I apply an ...
mavex857's user avatar
0 votes
1 answer
119 views

Why is PPO not choosing a solution that is giving a higher cumulative reward?

I use PPO to train my fermenter (digital twin) to maximize enzyme (product) production. action: 1 or 0 ie. add substrate at a particular time or not based on cell and enzymes present in the tank ...
user79474's user avatar
2 votes
1 answer
58 views

How does reward work while training a Reinforcement Learning agent?

I am using PPO to train my environment which I created using stable baselines 3. I am confused if I should make the reward = 0 in the step function or not. Initially, I used to have self.reward = 0 in ...
user79474's user avatar
0 votes
0 answers
104 views

PPO agent of path planning does not learn

I built an agent based on PPO with actor-critic structure for a project of path planning, in which the objective of agent is to find a path toward the goal bypass the obstacle in a 16*16 grid map. ...
speedhawk1's user avatar
0 votes
0 answers
28 views

What is the Input to the agent in the paper "Model Based Reinforcement Learning for Atari", and why does the world model run at inference time?

I am currently reading the paper Model based Reinforcement Learning for Atari. However, they do not specify what exactly they use as input for the agent. I believe it to be the observation space - ...
Nick's user avatar
  • 1
2 votes
0 answers
59 views

How does PPO with advantage normalization learn in MountainCar-v0 before first reaching the goal state?

I'm trying to figure out how PPO ever learns anything in a sparse environment like gymnasium's MountainCar-v0 before it first ever reaches the goal state. Specifically was looking at stable_baselines3'...
Switch's user avatar
  • 121
0 votes
0 answers
28 views

Does PPO clip kills gradient

In PPO Loss function, the reasoning to use the clip function is to do the update within a trusted region. But once we use the clip function, don't it kills the gradients with it? Because the only ...
Wenuka's user avatar
  • 111
0 votes
0 answers
35 views

Reinforcement Learning agent converging at the lowest reward

I have a custom environment created using Stable Baselines 3 where the environment is a digital twin of a fermentation reaction. It observes the enzyme activity which is the output of the fermentation ...
user79474's user avatar
1 vote
0 answers
16 views

Enhancing Generalization in DRL Agents in Static Data Environments

Context: I'm working with a deep reinforcement learning (DRL) agent in a market-like environment where its actions do not affect the environment. The environment uses historical data up to a certain ...
ElonMuskofBadIdeas's user avatar
0 votes
1 answer
1k views

Is PPO a policy-based method or an actor-critique-based method?

as far as i understand there are 3 categories of Reinforcement algorithms: Value-based methods (like DQN or Sarsa) Policy-based methods (like REINFORCE) Actor-critic-based methods (like A2C) To ...
PeterBe's user avatar
  • 256
0 votes
1 answer
61 views

In the PPO objective function, does the expectation do a weighted average over the batch? [closed]

I am currently working on implementing Proximal Policy Optimization (PPO) for a reinforcement learning task, and I have a few questions regarding the objective function and probability calculations. ...
TWTom's user avatar
  • 13
0 votes
0 answers
46 views

What algorithm should I use to train an antichess agent?

I will implement an antichess agent and am not sure about which algorithm to use. My current candidates are minimax with alpha-beta pruning, MCTS and proximal policy optimization. Should I consider ...
heyula's user avatar
  • 1
2 votes
0 answers
118 views

Why does only Deep Q Learning have an overestimation bias?

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were ...
Jerry Ding's user avatar
0 votes
1 answer
337 views

Understanding KL Stopping and KL Cutoff for the PPO algorithm

I am reading a couple of review papers to optimize the PPO algorithm. It seems like the review papers are saying the same thing but used slightly different terms. Could someone please tell if the ...
desert_ranger's user avatar
0 votes
1 answer
354 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
2 votes
1 answer
363 views

Filling replay buffer with expert trajectories for PPO/DQN

I have a reinforcement learning environment with sparse rewards. Current methods such as PPO and DQN both fail to learn a policy that is suffuciently good. I may have a way to find trajectories that ...
Erik Storm's user avatar
1 vote
0 answers
238 views

Negative KL-divergence RLHF implementation

I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace: What Is the Concern with Negative KL Divergence? If you generate text by purely ...
probably45's user avatar
1 vote
1 answer
233 views

How Can I Backpropagate My Network with PPO

I am trying to implement PPO to my reinforcement agents. I have a classic neural network that represents the policy. I didn't quite understand how the PPO updates the network, according to what? There ...
Ege's user avatar
  • 25
0 votes
0 answers
42 views

Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm

I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
Mohammad Hosseini's user avatar
0 votes
1 answer
1k views

How do I add Entropy to a PPO algorithm?

I learned about adding entropy to RL algorithms through the notes provided in SpinningUp. They explained how entropy is added to the SAC algorithm. Here is my understanding - In entropy regularized RL,...
desert_ranger's user avatar
0 votes
2 answers
159 views

Reinforcement learning: Find the fastest solution (minimal number of steps)

I have a Gym env (env) for which I train with a model using the PPO algorithm with stable-baselines. ...
P. Egli's user avatar
  • 101
2 votes
1 answer
194 views

Why do we limit the standard deviation in Actor architectures in Reinforcement Learning?

I'm in the process of implementing Actor-Critic structures for Reinforcement Learning (RL) and I've noticed that it's a common practice to limit the standard deviation (std). I've seen this in ...
XiaoBanni's user avatar
1 vote
1 answer
145 views

Why do we not calculate value for the done state in PPO?

In PPO a common pattern I see in calcualting advantages is: $$ delta = reward[t] + (gamma * valueNewState[t] * done[t]) - valueOldState[t]$$ Such as in this article. I am wondering why we multiply by <...
Jacob B's user avatar
  • 247
0 votes
0 answers
525 views

How does Proximal Policy Optimization deal with sparse reward

In the original paper, the objective of PPO is as follows:. My question is, how does this objective behave in a sparse reward setting (i.e., reward is only given after a sequence of actions were taken)...
Sam's user avatar
  • 195
0 votes
1 answer
68 views

How to expand an agent's action space with more actions?

I'm training a FPS agent using StableBaselines 3's PPO algorithm. To aid learning, I would like to train the agent using just a basic set of actions (e.g Turn left, turn right, shoot). After the agent ...
Ilija Vuk's user avatar
  • 103
1 vote
2 answers
251 views

What should I do, reinforcement learning agent gives different result on every train?

I'm using PPO+LSTM to create a trading bot. The agent is trained on 3 years of data and tested on 1 year. Every time I train the agent with same set of hyper-parameters, I get very different results ...
ad124j2's user avatar
  • 11
2 votes
1 answer
778 views

Should DQN/PPO be used for state spaces that are not that large?

I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal ...
gameveloster's user avatar
1 vote
1 answer
317 views

PPO: dealing with variable episodic length

I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
Antonis Karvelas's user avatar
2 votes
2 answers
254 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
fool's user avatar
  • 203
3 votes
4 answers
471 views

Why clip the PPO objective on only one side?

In PPO with clipped surrogate objective (see the paper here), we have the following objective: The shape of the function is shown in the image below, and depends on whether the advantage is positive ...
Jer's user avatar
  • 31
1 vote
1 answer
237 views

Is there a reward function that would encourage exploration in this case?

I am new to Reinforcement Learning. I am trying to train PPO agent for citylearn. The goal is to lower two environmental variables from observations. The default reward function is ...
Sai Dinesh Pola's user avatar
1 vote
1 answer
3k views

Reinforcement Learning with PPO - entropy loss dropping, but so is performance. Why?

I'm using PPO with an action-mask and I'm encountering a weird phenomenon. At first during training, the entropy loss is decreasing (I interpret this as less exploration, more exploitation, more "...
Vladimir Belik's user avatar
0 votes
1 answer
70 views

Why Phasic Policy Gradient (PPG) can update value function in auxiliary phase?

My questions is that how could we train the value network (separated from shared network) by using data from previous policies, which varies a lot since we collect data from different policies with ...
Magi Feeney's user avatar
0 votes
1 answer
113 views

Can TRPO use replay buffers?

I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$. Is it possible to ...
Wj210's user avatar
  • 1
1 vote
0 answers
83 views

How many parameters of a RL environment?

I’m working at a Reinforcement Learning model, using PPO algorithm, in which the agent has 4 possible actions, acting in a stochastic environment defined by 3 parameters. Given its stochasticity, I ...
Damuna Taliffato's user avatar
1 vote
1 answer
105 views

How to use RL on a robotic moving arm?

I'm working on a simulation of a motor that is attached to a wing (Later, this will also have a real-life counterpart once I'll assemble all the components in our lab), and I can control the forces/...
Hadar Sharvit's user avatar
1 vote
1 answer
893 views

Why does a PPO agent perform only the action that costs the least?

I am trying to implement an intelligent agent that can perform penetration testing within the nasim (link) environment, a network simulator. I would like to try to use parametric mode for actions, and ...
Francesco's user avatar
  • 133
1 vote
0 answers
92 views

How can I get an integer as output for continuous action space PPO reinforcement learning?

I have a huge discrete action space, the learning stability is not good. I'd like to move to continuous action space but the only output for my task can be a positive integer (let's say in the range 0 ...
NAnn's user avatar
  • 111
1 vote
0 answers
58 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
3 votes
1 answer
242 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
Peter's user avatar
  • 55
1 vote
1 answer
220 views

What would be the reason for having a different network architecture for the actor vs. value function networks in PPO?

I was reading this link , and saw some creative architectures for PPO. I know the "No Free Lunch Theorem" and all, but what would be the logic/reasoning for why you would choose to have a ...
Vladimir Belik's user avatar
3 votes
1 answer
8k views

Does SAC perform better than PPO in sample-expensive tasks with discrete action spaces?

I am currently using Proximal Policy Optimization (PPO) to solve my RL task. However, after reading about Soft Actor-Critic (SAC) now I am unsure whether I should stick to PPO or switch to SAC. ...
Aeryan's user avatar
  • 53