Questions tagged [proximal-policy-optimization]

For questions related to the reinforcement learning algorithm called proximal policy optimization (PPO), which was introduced in the paper "Proximal Policy Optimization Algorithms" (2017) by John Schulman et al.

Filter by
Sorted by
Tagged with
3
votes
0answers
60 views

Sampling in TRPO or PPO

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
1
vote
0answers
30 views

PPO: action std or entropy for exploration?

When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations : Exploration with action std : Collect trajectories on ...
1
vote
0answers
46 views

Is it possible to use Reward Function of type R(s, a, s') if more than one action is applied?

I am applying a reinforcement learning agent (PPO2, stable baselines implementation) to a custom built environment using OpenAI Gym. One reward function (formualted as loss function, that is, all ...
0
votes
0answers
40 views

Understanding policy update in PPO2

I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation: From the original paper I know that the policy parameters $\theta$ are updated K-...
1
vote
0answers
78 views

How to use the LSTM layer in PPO architecture?

What is the best way of using the LSTM layer in PPO architecture? Should I use them in the first layer of both actor and critic, or use them just before the final layer of these networks? Should I ...
1
vote
0answers
35 views

Understanding log probabilities of actions in PPO objective

I'm trying to implement Proximal Policy Optimization algorithm (code here) but am confused about certain concepts:- 1) What is the correct way to implement log probability of a policy (denoted by ...
0
votes
1answer
44 views

Entropy term in Proximal Policy Optimization (PPO) becomes undefined after few training epochs

I have implemented the total loss of my PPO objective as follows:- ...
2
votes
0answers
96 views

What is ratio of the objective function in the case of continuous action spaces?

I'm trying to implement the proximal policy optimization (PPO) algorithm. I'm confused on how to make it work with continuous action space. For discrete action space, the output of the network is the ...
1
vote
0answers
135 views

Implementation of PPO - Value Loss not converging, return plateauing

Copy from my reddit post: (Sorry if this does not fit here, please tell me and i delete it) Help regarding I'm working on an implementation of PPO, which i plan to use in my (Bachelors) Thesis. To ...
0
votes
0answers
33 views

How do I define the reward function in the case of self-driving raspberry pi car?

I am working on a self driving car powered by a raspberry pi. My first step is to use PPO to teach it to not run into walls. But I am having trouble getting it to work. I want to allow the car to ...
0
votes
1answer
174 views

Reinforcement learning with PPO: rewards decreasing

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
0
votes
1answer
33 views

Why don't we decorrelate transitions for policy-based data?

I'm implementing PPO myself strictly follow the steps: sample transitions randomly shuffle the sampled transitions compute gradients and update networks using the sampled transitions drop transitions ...
0
votes
1answer
304 views

Getting NaN from A3C PPO model [closed]

I've pieced together this A3C w/ PPO Gym Pendulum example, but I'm finding after a while, when attempting to get the action from the model, I get a NaN return: ...
2
votes
1answer
372 views

How do I calculate the policy in the Proximal Policy Optimization algorithm?

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are ...
5
votes
2answers
514 views

Why is the log probability replaced with the importance sampling in the loss function?

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients $$...