Questions tagged [policy-gradients]

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

Filter by
Sorted by
Tagged with
0 votes
2 answers
52 views

In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?

Taking as an example the Advantage Actor Critic, the objective function is: \begin{equation} \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
user avatar
0 votes
0 answers
18 views

Can A2C deal with a reward that is decided later than action selection?

I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
user77436's user avatar
0 votes
1 answer
51 views

Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
Ufuk Can Bicici's user avatar
3 votes
0 answers
51 views

Why policy gradient theorem has two different forms?

I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
Yuxiang Wei's user avatar
0 votes
0 answers
24 views

Policy Gradient in Partial Observability

Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
A J's user avatar
  • 21
0 votes
1 answer
106 views

Is my PPO agent learning? or is it just exploring?

I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ...
GPra's user avatar
  • 1
0 votes
0 answers
29 views

actor critic gradient inconsistent with mathematical proofs

From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as: $$\bigtriangledown_\theta \eta(\...
Hooman's user avatar
  • 1
0 votes
0 answers
41 views

How to setup correctly a sequence generation task with RL/policy gradient learning?

I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points. So, I have the following model and loss function: ...
eris's user avatar
  • 1
0 votes
0 answers
41 views

Updating custom output layers of an LSTM network

I have a text generation task learning to predict the next word with an LSTM network with multiple output layers. After the generation of a sentence has finished, I calculate a reward for the whole ...
eris's user avatar
  • 1
3 votes
1 answer
167 views

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
cfml's user avatar
  • 53
1 vote
1 answer
60 views

Zero reward in policy gradient

Specifically, according to this post: How is the policy gradient calculated in REINFORCE the function I need to minimise is: $−Gt \log \pi(At|St,θt)$ where $Gt$ is the discounted reward, and $\pi$ is ...
Jason L's user avatar
  • 111
1 vote
1 answer
109 views

How do policy gradients work?

If I understand it correctly from the following equation $$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$ ...
User's user avatar
  • 175
2 votes
1 answer
103 views

Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents

Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
Luca Anzalone's user avatar
1 vote
1 answer
130 views

Is the re-parameterization trick necessary in the policy gradient method?

If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action. Wouldn't this lead to the same issue that variational autoencoders ...
Sam's user avatar
  • 175
0 votes
0 answers
81 views

How to increase exploration in IMPALA?

I am using the RLLib library for applying IMPALA algorithm to my environment. I want to know which variables in https://docs.ray.io/en/latest/_modules/ray/rllib/algorithms/impala/impala.html#...
Saeid Ghafouri's user avatar
3 votes
1 answer
157 views

Is it always a good idea to use deterministic policies during testing?

I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ...
desert_ranger's user avatar
5 votes
2 answers
154 views

Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
jwl17's user avatar
  • 59
0 votes
1 answer
325 views

What is the problem in my implementation of actor critic?

I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1". As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
Labo's user avatar
  • 111
0 votes
0 answers
24 views

Explain policy gradient proof from reinforcement learning book [duplicate]

How is this transition below happening? I don't understand. Page 325 of rl book http://www.incompleteideas.net/book/RLbook2020.pdf
Andriy Drozdyuk's user avatar
1 vote
1 answer
45 views

Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?

Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number ...
Archie Gertsman's user avatar
0 votes
0 answers
80 views

Why do policy gradient methods not work for imperfect information games?

I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't ...
JacKeown's user avatar
  • 125
1 vote
1 answer
51 views

How can I formulate a usecase with an additional constraint as a reinforment learning problem?

I am new to the field of reinforcement learning, and I feel a recent use case of mine is highly relevant, but I don't know how to forumate it as a typical reinforcement learning problem. Let's say I ...
Jim Wang's user avatar
  • 111
4 votes
1 answer
377 views

Why are policy gradient methods more effective in high-dimensional action spaces?

David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
Saucy Goat's user avatar
1 vote
0 answers
62 views

Reinforcement Learning - Independence between current state and future state

I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ...
MaarcosNascimen's user avatar
2 votes
2 answers
135 views

Where does the proximal policy optimization objective's ratio term come from?

I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$? Put ...
user's user avatar
  • 203
2 votes
1 answer
232 views

What makes TRPO an actor-critic method? Where is the critic?

From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
thesofakillers's user avatar
0 votes
1 answer
37 views

Why are policy gradients popular in RL when there exists a dual LP formulation in terms of occupation measures that can be solved easily?

Why are policy gradient methods popular in reinforcement learning when there exists a dual LP formulation in terms of occupation measures that can be solved easily?
blackbird_h71's user avatar
1 vote
0 answers
42 views

Policy Gradient Methods when using a fixed initial sequence of actions

I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question. In my ...
Ryan Keathley's user avatar
1 vote
0 answers
60 views

Understanding the features given in Example 13.1 of Sutton and Barto

I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book. I assume as it is a free pdf ...
topher217's user avatar
  • 111
0 votes
0 answers
64 views

How do I improve the reward of policy gradient network when multiple states and actions exist per time step?

I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets. The details are as follows. ...
aby's user avatar
  • 1
1 vote
0 answers
96 views

In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?

In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting ...
Davaasuren Nyamdavaa's user avatar
0 votes
1 answer
122 views

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
Josh's user avatar
  • 89
0 votes
1 answer
80 views

How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper?

The paper is here https://arxiv.org/pdf/1906.00190.pdf and the relevant paragraph where they explain their method is below: It's still not clear to me how this is meant to work exactly. In the pseudo-...
Ryan Keathley's user avatar
3 votes
1 answer
109 views

Why might my policy gradient agent appear to maximize the absolute value of rewards?

I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
Josh's user avatar
  • 89
2 votes
0 answers
48 views

For REINFORCE, why do different books give different algorithms?

The discount rate appears twice in the REINFORCE algorithm in Sutton and Barto (2018). However, in three major books (Graesser and Keng (2020); Morales (2020); Ravichandiran (2020)) on reinforcement ...
Averill M. Law's user avatar
2 votes
1 answer
472 views

In the Policy Gradient Theorem proof, why is $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$ true?

I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below: I don't understand how the last ...
Fady's user avatar
  • 31
2 votes
1 answer
57 views

Comparing Reinforcement Learning models

I am currently completing my thesis on optimising combinatorial problems, and we decided to utilize reinforcement learning. The problem is that I am not sure which algorithm to choose. Is there a ...
Rami Hoteit's user avatar
1 vote
0 answers
50 views

Can entropy bonus be used with state-independent log std for stochastic policies?

In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
flxh's user avatar
  • 131
4 votes
0 answers
155 views

What is the correct interpretation of the discount factor in MDPs?

In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
Skander's user avatar
  • 41
3 votes
1 answer
201 views

Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
Peter's user avatar
  • 55
2 votes
0 answers
50 views

Can objective function and gradient be unlimited in reinforcement learning?

I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
pippo's user avatar
  • 41
4 votes
2 answers
301 views

How do we estimate the value of a stochastic policy?

I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
mac_or_cheese's user avatar
1 vote
0 answers
62 views

Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)

I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
Penn's user avatar
  • 11
0 votes
1 answer
53 views

Does the policy search work if there is no state to state dependency through actions?

There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
veerendra's user avatar
1 vote
0 answers
237 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
Kronic's user avatar
  • 49
1 vote
1 answer
375 views

Does importance sampling really improve sampling efficiency of TRPO or PPO?

Vanilla policy gradient has a loss function: $$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$ while in TRPO it is: $$\mathcal{L}_{\pi_{\...
Magi Feeney's user avatar
0 votes
2 answers
366 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
Taw's user avatar
  • 1,201
1 vote
1 answer
324 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
hanugm's user avatar
  • 3,612
2 votes
0 answers
105 views

In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?

In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated actions [may be] chosen from a normal (Gaussian) distribution. However, I ...
Bennet Leff's user avatar
1 vote
0 answers
63 views

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay

In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration like this : ...
parradox's user avatar