Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [policy-gradients]

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

in the "reward to go" trick in policy gradient methods, I have a question about the proof?

I am specifically talking about this proof Why does the "reward to go" trick in policy gradient methods work? where Dennis says 'on ith iteration the outer sum of random variable and ...
abhilash sharma's user avatar
0 votes
0 answers
43 views

Reinforcement learning - calculating policy gradient using cross entropy loss

I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/...
Blato's user avatar
  • 1
2 votes
1 answer
79 views

Should I use RL for this allocation problem and, if yes, which RL approach?

I want to use RL to solve the problem of Unit Allocation. SETTING: So for example in a game you can use units of type A and units of type B and you know that your opposing player has units of type C ...
Astraeus's user avatar
0 votes
0 answers
25 views

Does PPO clip kills gradient

In PPO Loss function, the reasoning to use the clip function is to do the update within a trusted region. But once we use the clip function, don't it kills the gradients with it? Because the only ...
Wenuka's user avatar
  • 111
2 votes
1 answer
66 views

Is $s_0$ the current state in policy gradients?

As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is: $$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$ where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ ...
fermented_bean's user avatar
2 votes
1 answer
44 views

In policy gradient algorithms, if the model always predicts any action with a probability of 1, will the gradient always be 0?

The loss of policy gradient: $$ \nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi\theta}\left[\sum_{t=0}^{T}(R_t-b_t)\nabla_{\theta}\log\pi_\theta(a_t|s_t)\right] $$ where $\pi_\theta(a_t|s_t)$ is the ...
Devymex's user avatar
  • 123
0 votes
2 answers
81 views

In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?

Taking as an example the Advantage Actor Critic, the objective function is: \begin{equation} \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
user avatar
0 votes
0 answers
20 views

Can A2C deal with a reward that is decided later than action selection?

I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
user77436's user avatar
0 votes
1 answer
72 views

Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
Ufuk Can Bicici's user avatar
4 votes
0 answers
106 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
0 answers
25 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 21
0 votes
1 answer
272 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
0 votes
0 answers
29 views

actor critic gradient inconsistent with mathematical proofs

From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as: $$\bigtriangledown_\theta \eta(\...
Hooman's user avatar
  • 1
0 votes
0 answers
45 views

How to setup correctly a sequence generation task with RL/policy gradient learning?

I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points. So, I have the following model and loss function: ...
eris's user avatar
  • 1
0 votes
0 answers
42 views

Updating custom output layers of an LSTM network

I have a text generation task learning to predict the next word with an LSTM network with multiple output layers. After the generation of a sentence has finished, I calculate a reward for the whole ...
eris's user avatar
  • 1
3 votes
2 answers
257 views

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
cfml's user avatar
  • 53
1 vote
1 answer
99 views

Zero reward in policy gradient

Specifically, according to this post: How is the policy gradient calculated in REINFORCE the function I need to minimise is: $−Gt \log \pi(At|St,θt)$ where $Gt$ is the discounted reward, and $\pi$ is ...
Jason L's user avatar
  • 111
1 vote
1 answer
133 views

How do policy gradients work?

If I understand it correctly from the following equation $$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$ ...
User's user avatar
  • 175
2 votes
1 answer
149 views

Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
Luca Anzalone's user avatar
1 vote
1 answer
196 views

Is the re-parameterization trick necessary in the policy gradient method?

If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action. Wouldn't this lead to the same issue that variational autoencoders ...
Sam's user avatar
  • 195
3 votes
1 answer
223 views

Is it always a good idea to use deterministic policies during testing?

I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ...
desert_ranger's user avatar
5 votes
2 answers
167 views

Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
jwl17's user avatar
  • 59
1 vote
1 answer
469 views

What is the problem in my implementation of actor critic?

I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1". As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
Labo's user avatar
  • 121
1 vote
1 answer
45 views

Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?

Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number ...
Archie Gertsman's user avatar
0 votes
0 answers
88 views

Why do policy gradient methods not work for imperfect information games?

I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't ...
JacKeown's user avatar
  • 125
1 vote
1 answer
51 views

How can I formulate a usecase with an additional constraint as a reinforment learning problem?

I am new to the field of reinforcement learning, and I feel a recent use case of mine is highly relevant, but I don't know how to forumate it as a typical reinforcement learning problem. Let's say I ...
Jim Wang's user avatar
  • 111
4 votes
1 answer
470 views

Why are policy gradient methods more effective in high-dimensional action spaces?

David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
Saucy Goat's user avatar
1 vote
0 answers
70 views

Reinforcement Learning - Independence between current state and future state

I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ...
MaarcosNascimen's user avatar
2 votes
2 answers
221 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
fool's user avatar
  • 203
2 votes
1 answer
302 views

What makes TRPO an actor-critic method? Where is the critic?

From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
thesofakillers's user avatar
0 votes
1 answer
47 views

Why are policy gradients popular in RL when there exists a dual LP formulation in terms of occupation measures that can be solved easily?

Why are policy gradient methods popular in reinforcement learning when there exists a dual LP formulation in terms of occupation measures that can be solved easily?
blackbird_h71's user avatar
1 vote
0 answers
46 views

Policy Gradient Methods when using a fixed initial sequence of actions

I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question. In my ...
Ryan Keathley's user avatar
1 vote
0 answers
84 views

Understanding the features given in Example 13.1 of Sutton and Barto

I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book. I assume as it is a free pdf ...
topher217's user avatar
  • 111
0 votes
0 answers
69 views

How do I improve the reward of policy gradient network when multiple states and actions exist per time step?

I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets. The details are as follows. ...
aby's user avatar
  • 1
1 vote
0 answers
122 views

In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?

In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting ...
Davaasuren Nyamdavaa's user avatar
0 votes
1 answer
145 views

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
Josh's user avatar
  • 89
0 votes
1 answer
87 views

How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper?

The paper is here https://arxiv.org/pdf/1906.00190.pdf and the relevant paragraph where they explain their method is below: It's still not clear to me how this is meant to work exactly. In the pseudo-...
Ryan Keathley's user avatar
3 votes
1 answer
142 views

Why might my policy gradient agent appear to maximize the absolute value of rewards?

I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
Josh's user avatar
  • 89
2 votes
0 answers
55 views

For REINFORCE, why do different books give different algorithms?

The discount rate appears twice in the REINFORCE algorithm in Sutton and Barto (2018). However, in three major books (Graesser and Keng (2020); Morales (2020); Ravichandiran (2020)) on reinforcement ...
Averill M. Law's user avatar
2 votes
1 answer
693 views

In the Policy Gradient Theorem proof, why is $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$ true?

I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below: I don't understand how the last ...
Fady's user avatar
  • 31
2 votes
1 answer
60 views

Comparing Reinforcement Learning models

I am currently completing my thesis on optimising combinatorial problems, and we decided to utilize reinforcement learning. The problem is that I am not sure which algorithm to choose. Is there a ...
Rami Hoteit's user avatar
1 vote
0 answers
56 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
4 votes
0 answers
186 views

What is the correct interpretation of the discount factor in MDPs?

In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
Skander's user avatar
  • 41
3 votes
1 answer
234 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
Peter's user avatar
  • 55
2 votes
0 answers
50 views

Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
pippo's user avatar
  • 41
4 votes
2 answers
393 views

How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
mac_or_cheese's user avatar
1 vote
0 answers
64 views

Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
Penn's user avatar
  • 11
0 votes
1 answer
62 views

Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
veerendra's user avatar
1 vote
0 answers
256 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
Kronic's user avatar
  • 49
1 vote
1 answer
484 views

Does importance sampling really improve sampling efficiency of TRPO or PPO?

Vanilla policy gradient has a loss function: $$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$ while in TRPO it is: $$\mathcal{L}_{\pi_{\...
Magi Feeney's user avatar