Questions tagged [policy-gradients]
For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.
194
questions
0
votes
2
answers
52
views
In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate?
Taking as an example the Advantage Actor Critic, the objective function is:
\begin{equation}
\nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\...
user77931
0
votes
0
answers
18
views
Can A2C deal with a reward that is decided later than action selection?
I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ...
0
votes
1
answer
51
views
Discount factors in REINFORCE Algorithm: Difference in two definitions
I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/):
Here, at every step in an ...
- 101
3
votes
0
answers
51
views
Why policy gradient theorem has two different forms?
I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &...
- 31
0
votes
0
answers
24
views
Policy Gradient in Partial Observability
Let $\pi_{\theta}$ be a policy. Then, I was able to follow through the proof of: $\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ...
- 21
0
votes
1
answer
106
views
Is my PPO agent learning? or is it just exploring?
I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src.
My doubts are mainly due ...
- 1
0
votes
0
answers
29
views
actor critic gradient inconsistent with mathematical proofs
From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to $\theta$ with baseline is proved as:
$$\bigtriangledown_\theta \eta(\...
- 1
0
votes
0
answers
41
views
How to setup correctly a sequence generation task with RL/policy gradient learning?
I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points.
So, I have the following model and loss function:
...
- 1
0
votes
0
answers
41
views
Updating custom output layers of an LSTM network
I have a text generation task learning to predict the next word with an LSTM network with multiple output layers.
After the generation of a sentence has finished, I calculate a reward for the whole ...
- 1
3
votes
1
answer
167
views
$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)
I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition :
Generalize the box on page 199, the policy gradient theorem (13.5),
the proof of the policy ...
- 53
1
vote
1
answer
60
views
Zero reward in policy gradient
Specifically, according to this post:
How is the policy gradient calculated in REINFORCE
the function I need to minimise is:
$−Gt \log \pi(At|St,θt)$
where $Gt$ is the discounted reward, and $\pi$ is ...
- 111
1
vote
1
answer
109
views
How do policy gradients work?
If I understand it correctly from the following equation
$$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$
...
- 175
2
votes
1
answer
103
views
Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents
Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ...
- 2,443
1
vote
1
answer
130
views
Is the re-parameterization trick necessary in the policy gradient method?
If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action.
Wouldn't this lead to the same issue that variational autoencoders ...
- 175
0
votes
0
answers
81
views
How to increase exploration in IMPALA?
I am using the RLLib library for applying IMPALA algorithm to my environment. I want to know which variables in https://docs.ray.io/en/latest/_modules/ray/rllib/algorithms/impala/impala.html#...
- 123
3
votes
1
answer
157
views
Is it always a good idea to use deterministic policies during testing?
I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ...
- 608
5
votes
2
answers
154
views
Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?
In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
- 59
0
votes
1
answer
325
views
What is the problem in my implementation of actor critic?
I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1".
As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
- 111
0
votes
0
answers
24
views
Explain policy gradient proof from reinforcement learning book [duplicate]
How is this transition below happening? I don't understand.
Page 325 of rl book
http://www.incompleteideas.net/book/RLbook2020.pdf
- 101
1
vote
1
answer
45
views
Could one still learn a good policy by doing a backprop every fixed number of steps within an episode?
Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number ...
- 121
0
votes
0
answers
80
views
Why do policy gradient methods not work for imperfect information games?
I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't ...
- 125
1
vote
1
answer
51
views
How can I formulate a usecase with an additional constraint as a reinforment learning problem?
I am new to the field of reinforcement learning, and I feel a recent use case of mine is highly relevant, but I don't know how to forumate it as a typical reinforcement learning problem.
Let's say I ...
- 111
4
votes
1
answer
377
views
Why are policy gradient methods more effective in high-dimensional action spaces?
David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ...
- 143
1
vote
0
answers
62
views
Reinforcement Learning - Independence between current state and future state
I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ...
2
votes
2
answers
135
views
Where does the proximal policy optimization objective's ratio term come from?
I will use the notation used in the proximal policy optimization paper.
What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?
Put ...
- 203
2
votes
1
answer
232
views
What makes TRPO an actor-critic method? Where is the critic?
From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size $\beta$ from a KL constraint between the new and ...
- 309
0
votes
1
answer
37
views
Why are policy gradients popular in RL when there exists a dual LP formulation in terms of occupation measures that can be solved easily?
Why are policy gradient methods popular in reinforcement learning when there exists a dual LP formulation in terms of occupation measures that can be solved easily?
1
vote
0
answers
42
views
Policy Gradient Methods when using a fixed initial sequence of actions
I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question.
In my ...
1
vote
0
answers
60
views
Understanding the features given in Example 13.1 of Sutton and Barto
I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book.
I assume as it is a free pdf ...
- 111
0
votes
0
answers
64
views
How do I improve the reward of policy gradient network when multiple states and actions exist per time step?
I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets.
The details are as follows.
...
- 1
1
vote
0
answers
96
views
In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach?
In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting ...
0
votes
1
answer
122
views
What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
- 89
0
votes
1
answer
80
views
How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper?
The paper is here https://arxiv.org/pdf/1906.00190.pdf
and the relevant paragraph where they explain their method is below:
It's still not clear to me how this is meant to work exactly. In the pseudo-...
3
votes
1
answer
109
views
Why might my policy gradient agent appear to maximize the absolute value of rewards?
I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
- 89
2
votes
0
answers
48
views
For REINFORCE, why do different books give different algorithms?
The discount rate appears twice in the REINFORCE algorithm in Sutton and Barto (2018). However, in three major books (Graesser and Keng (2020); Morales (2020); Ravichandiran (2020)) on reinforcement ...
2
votes
1
answer
472
views
In the Policy Gradient Theorem proof, why is $d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)$ true?
I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below:
I don't understand how the last ...
- 31
2
votes
1
answer
57
views
Comparing Reinforcement Learning models
I am currently completing my thesis on optimising combinatorial problems, and we decided to utilize reinforcement learning. The problem is that I am not sure which algorithm to choose. Is there a ...
- 31
1
vote
0
answers
50
views
Can entropy bonus be used with state-independent log std for stochastic policies?
In this blog article by openai, they say the std of the exploration distribution must be state-dependent, i.e. an output of the policy network, so it works with the entropy bonus, which is an integral ...
- 131
4
votes
0
answers
155
views
What is the correct interpretation of the discount factor in MDPs?
In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
- 41
3
votes
1
answer
201
views
Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?
I understand how to derive the vanilla policy gradient
$$
\begin{align}
\nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \...
- 55
2
votes
0
answers
50
views
Can objective function and gradient be unlimited in reinforcement learning?
I'm looking at an example where they define a policy $\pi_\theta(a_t|s_t)\sim \mathcal{N}(ks_t, \sigma)$, where $a_t$ and $s_t$ are action and state, while $\theta=(k,\sigma)$ are the parameters of ...
- 41
4
votes
2
answers
301
views
How do we estimate the value of a stochastic policy?
I'm learning about reinforcement learning, particularly policy gradient methods and actor-critic methods. I've noticed that many algortihms use stochastic policies during training (i.e. they select ...
1
vote
0
answers
62
views
Policy gradient (or more general, RL algorithms) for the problems where actions does not determine next state (next state is independent to action)
I am pretty new in RL. Could anyone suggest results/paper about whether or not policy gradient (or more general RL algorithms) can be applied to the problems where actions does not determine next ...
- 11
0
votes
1
answer
53
views
Does the policy search work if there is no state to state dependency through actions?
There is a game in which the state comes one after the other without depending on the agent's action. The agent gets a reward for its actions at the end of the game. The goal of the agent is to reach ...
- 19
1
vote
0
answers
237
views
Why is an action-independent baseline required to reduce variance?
I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
- 49
1
vote
1
answer
375
views
Does importance sampling really improve sampling efficiency of TRPO or PPO?
Vanilla policy gradient has a loss function:
$$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$
while in TRPO it is:
$$\mathcal{L}_{\pi_{\...
- 41
0
votes
2
answers
366
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 1,201
1
vote
1
answer
324
views
Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
Consider the following algorithm from the textbook titled Reinforcement
Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart
While playing the game for the generation of ...
- 3,612
2
votes
0
answers
105
views
In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?
In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated
actions [may be] chosen from a normal (Gaussian) distribution.
However, I ...
- 121
1
vote
0
answers
63
views
Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay
In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration
like this : ...
- 11