Questions tagged [policy-gradients]

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

13 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
0answers
36 views

Sampling in TRPO or PPO

In the TRPO paper, the objective to maximize is (equation 14) $$ \mathbb{E}_{s\sim\rho_{\theta_\text{old}},a\sim q}\left[\frac{\pi_\theta(a|s)}{q(a|s)} Q_{\theta_\text{old}}(s,a) \right] $$ which ...
2
votes
0answers
55 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
2
votes
0answers
33 views

How does the TRPO surrogate loss account for the error in the policy?

In the Trust Region Policy Optimization (TRPO) paper, on page 10, it is stated An informal overview is as follows. Our proof relies on the notion of coupling, where we jointly define the ...
2
votes
0answers
71 views

Can I use deterministic policy gradient methods for stochastic policy learning?

Can I treat a stochastic policy (over a finite action space of size $n$) as a deterministic policy (in the set of probability distribution in $\mathbb{R}^n$)? It seems to me that nothing is broken ...
2
votes
0answers
29 views

How to include exploration in Gaussian policy

When dealing with continuous action spaces, a common choice when designing a policy in policy gradient methods is to learn mean and variance of actions for a specific state and then simply sample from ...
1
vote
0answers
22 views

PPO: action std or entropy for exploration?

When trying to implement my own PPO (Proximal Policy Optimizer), I came accross two different implementations : Exploration with action std : Collect trajectories on ...
1
vote
0answers
68 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
1
vote
0answers
23 views

Impact of Varying Length Trajectories on Policy Gradient Optimization

As the question states, I am wondering how, if at all, a varying length of a trajectory (series of state,action pairs) will impact training/performance of policy gradient algorithms such as PPO, TRPO ...
0
votes
0answers
32 views

Understanding policy update in PPO2

I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation: From the original paper I know that the policy parameters $\theta$ are updated K-...
0
votes
0answers
52 views

Encoding real valued inputs

UPDATE: After reading more about the topic, I've tried implementing the DDPG algorithm instead of using a variation of Q-Learning and still have the same issue. I have the following issue: I want ...
0
votes
1answer
121 views

Reinforcement learning with PPO: rewards decreasing

I'm trying to train a PPO agent and my average rewards graph looks like this. Could this indicate that it's stuck at a local maximum? Do I need to promote exploring by increasing the entropy or does ...
0
votes
0answers
114 views

Policy gradient in keras predicts only one action

I have trouble with the REINFORCE algorithm in keras with Atari games. After round about 30 episodes the network converges to one action. But the same algorithm is working with CartPole-v1 and ...
0
votes
1answer
48 views

Model-based Reinforcement Learning algorithm for real-time robotics task

I'm quite a newbie when it comes to practically working with Deep Learning techniques, although I studied them quite a lot theoretically in the last months. However, now I'm facing my first practical ...