The 2024 Developer Survey results are live! See the results

For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.

206 questions
Filter by
Sorted by
Tagged with
392 views

If I understand correctly, the goal of vanilla policy gradients is maximizing $E[r(s_t,a_t);\pi_\theta]$; in deriving the gradient of this function as a clearer function on $\theta$, we get $\sum_{t=0}... • 215 0 votes 1 answer 34 views I have a few doubts understanding and implementing Proximal Policy Optimisation Algorithm [closed] What is the difference between a rollout buffer and a replay buffer (as used in DQNs). Why can't they be used interchangeably? Why is the trajectory sampling parallelized? Is it just for making data ... 1 vote 2 answers 81 views How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm: How are these terms equivalent, and ... 2 votes 1 answer 74 views Why do big policy updates cause performance drop in deep RL? In the TRPO and PPO papers, it is mentioned that large policy updates often lead to performance drops in policy gradient methods. By "large policy updates," they mean a significant KL ... • 171 0 votes 1 answer 33 views Would the DDPG algorithm still function effectively if some transitions stored in its replay buffer are generated by a completely unrelated policy? Let's hypothesize a scenario where some of the records (si, ai, ri, si+1) in the replay buffer are generated by another completely unrelated random policy. If the DDPG algorithm still samples random ... 0 votes 1 answer 95 views Why is policy gradient theorem so important? What is the problem that the policy gradient solves? From what I understand the problem is taking the gradient of the state distirbution$d^{\pi_{\theta}}$, but what is exactly the problem here (maybe ... • 125 1 vote 1 answer 73 views Proof of gradient of$v_{\pi}(s)$via Kronecker Product Hi I am reading Mathematical Foundation of Reinforcement Learning by Shiyu Zhao and I try to understand a proof regarding policy gradients. The part is on page 209/210 in Policy Gradient Methods. ... • 125 0 votes 1 answer 48 views Does the off-policy/on-policy regime plays role in Contextual Bandits? I would like to bring up a point regarding the application of on-policy algorithms, such as REINFORCE, to contextual bandit problems using data collected from other policies. Here are my thoughts: In ... • 177 0 votes 0 answers 20 views in the "reward to go" trick in policy gradient methods, I have a question about the proof? I am specifically talking about this proof Why does the "reward to go" trick in policy gradient methods work? where Dennis says 'on ith iteration the outer sum of random variable and ... 0 votes 0 answers 58 views Reinforcement learning - calculating policy gradient using cross entropy loss I am writing a program that uses reinforcement learning and the policy gradient method to play Pong. It basically extends Andrej Karpathy's version (https://gist.github.com/karpathy/... 2 votes 1 answer 82 views Should I use RL for this allocation problem and, if yes, which RL approach? I want to use RL to solve the problem of Unit Allocation. SETTING: So for example in a game you can use units of type A and units of type B and you know that your opposing player has units of type C ... • 23 0 votes 0 answers 29 views Does PPO clip kills gradient In PPO Loss function, the reasoning to use the clip function is to do the update within a trusted region. But once we use the clip function, don't it kills the gradients with it? Because the only ... • 111 2 votes 1 answer 72 views Is$s_0$the current state in policy gradients? As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is: $$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$ where$R(\tau)=r_0+r_1+...+r_T$, with$r_t$... 2 votes 1 answer 47 views In policy gradient algorithms, if the model always predicts any action with a probability of 1, will the gradient always be 0? The loss of policy gradient: $$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi\theta}\left[\sum_{t=0}^{T}(R_t-b_t)\nabla_{\theta}\log\pi_\theta(a_t|s_t)\right]$$ where$\pi_\theta(a_t|s_t)$is the ... • 123 0 votes 2 answers 93 views In policy gradient methods why do we compute the gradient of the objective function through a one-trajectory estimate? Taking as an example the Advantage Actor Critic, the objective function is: \nabla_{\boldsymbol{\theta}} J(\boldsymbol{\theta})=\mathbb{E}_{\tau \sim \pi_{\boldsymbol{\theta}}}\left[\... 0 votes 0 answers 23 views Can A2C deal with a reward that is decided later than action selection? I am trying to use a policy gradient based RL algorithm, A2C. However, my training case is slightly different from what typical training tragets are. In my case, a reward is given not immediately ... 0 votes 1 answer 79 views Discount factors in REINFORCE Algorithm: Difference in two definitions I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ... 4 votes 0 answers 131 views Why policy gradient theorem has two different forms? I have been studying policy gradients recently but found different expositions from different sources, which greatly confused me. From the book "Reinforcement Learning: an Introduction (Sutton &... 0 votes 0 answers 26 views Policy Gradient in Partial Observability Let$\pi_{\theta}$be a policy. Then, I was able to follow through the proof of:$\nabla_\theta J=\mathbb{E}_{\tau\sim\pi_{\theta}}[\Sigma_{i=1}^T \nabla_\theta log(p_{\theta}(a_i|s_i)R(\tau)]$, where ... • 21 0 votes 1 answer 371 views Is my PPO agent learning? or is it just exploring? I implemented from scratch PPO to solve a custom RL environment. If you want, you can check the code here https://github.com/GiacomoPracucci/RL-edge-computing/tree/main/src. My doubts are mainly due ... • 1 0 votes 0 answers 31 views actor critic gradient inconsistent with mathematical proofs From different books and the courses (including Sutton and Barto, page 332 and cs229 notes, page 222, the gradient with respect to$\theta$with baseline is proved as: $$\bigtriangledown_\theta \eta(\... 0 votes 0 answers 46 views How to setup correctly a sequence generation task with RL/policy gradient learning? I've a pretrained model for sequence generation that I'd like to improve with RL but there are several shady points. So, I have the following model and loss function: ... • 1 0 votes 0 answers 42 views Updating custom output layers of an LSTM network I have a text generation task learning to predict the next word with an LSTM network with multiple output layers. After the generation of a sentence has finished, I calculate a reward for the whole ... • 1 3 votes 2 answers 282 views \gamma^t in REINFORCE update (Sutton-Barto RL book Exercise 13.2) I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ... • 53 1 vote 1 answer 117 views Zero reward in policy gradient Specifically, according to this post: How is the policy gradient calculated in REINFORCE the function I need to minimise is: −Gt \log \pi(At|St,θt) where Gt is the discounted reward, and \pi is ... • 111 1 vote 1 answer 142 views How do policy gradients work? If I understand it correctly from the following equation$$U(\theta)=\mathbb{E}_{\tau \sim P(\tau;\theta)}\left [ \sum_{t=0}^{H-1}R(s_t,u_t);\pi_{\theta} \right ]=\sum_{\tau}P(\tau;\theta)R(\tau)$$... • 215 2 votes 1 answer 163 views Mean or Mode of Action Distribution when Evaluating Policy Gradient Agents Policy gradient agents like A2C, PPO, etc learn a distribution over the action space that is parametrized by a neural net. For continuous actions the distribution is usually a Gaussian, while for ... • 3,046 1 vote 1 answer 218 views Is the re-parameterization trick necessary in the policy gradient method? If we want to learn a stochastic policy with the policy gradient method, we have to sample from the distribution to get an action. Wouldn't this lead to the same issue that variational autoencoders ... • 195 3 votes 1 answer 241 views Is it always a good idea to use deterministic policies during testing? I frequently see people setting deterministic = True while testing an RL algorithm. But is this the right approach? For instance, what happens if the agent plays rock, paper, and scissors? In this ... 5 votes 2 answers 181 views Why is$\sum_{s} \eta(s)$a constant of proportionality in the proof of the policy gradient theorem? In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ... • 59 1 vote 1 answer 509 views What is the problem in my implementation of actor critic? I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1". As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://... • 121 1 vote 1 answer 45 views Could one still learn a good policy by doing a backprop every fixed number of steps within an episode? Waiting an entire episode before doing a backprop can build up a very large computational graph, which is a burden on memory. Could one still learn a good policy by doing a backprop every fixed number ... 0 votes 0 answers 90 views Why do policy gradient methods not work for imperfect information games? I've heard before that policy gradient and Q-learning approaches fail on games of imperfect information. I was watching this video (starting at 23:45) about the Player of Games AI, but I couldn't ... • 125 1 vote 1 answer 52 views How can I formulate a usecase with an additional constraint as a reinforment learning problem? I am new to the field of reinforcement learning, and I feel a recent use case of mine is highly relevant, but I don't know how to forumate it as a typical reinforcement learning problem. Let's say I ... • 111 4 votes 1 answer 499 views Why are policy gradient methods more effective in high-dimensional action spaces? David Silver argues, in his Reinforcement Learning course, that policy-based reinforcement learning (RL) is more effective than value-based RL in high-dimensional action spaces. He points out that the ... • 143 1 vote 0 answers 71 views Reinforcement Learning - Independence between current state and future state I'm working on a real problem with continuous large actions range, where my agent takes actions based only on the current state of the environment, transitioning to a future state that is unrelated to ... 2 votes 2 answers 261 views Where does the proximal policy optimization objective's ratio term come from? I will use the notation used in the proximal policy optimization paper. What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio$r_t(\theta)$? Put ... • 203 2 votes 1 answer 331 views What makes TRPO an actor-critic method? Where is the critic? From what I understand, Trust Region Policy Optimization (TRPO) is a modification on Natural Policy Gradient (NPG) that derives the optimal step size$\beta$from a KL constraint between the new and ... 0 votes 1 answer 55 views Why are policy gradients popular in RL when there exists a dual LP formulation in terms of occupation measures that can be solved easily? Why are policy gradient methods popular in reinforcement learning when there exists a dual LP formulation in terms of occupation measures that can be solved easily? 1 vote 0 answers 48 views Policy Gradient Methods when using a fixed initial sequence of actions I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question. In my ... 1 vote 0 answers 94 views Understanding the features given in Example 13.1 of Sutton and Barto I'm struggling to understand the notation used to represent the features within Example 13.1 (Short corridor with switched actions" in the Sutton and Barto RL book. I assume as it is a free pdf ... • 111 0 votes 0 answers 70 views How do I improve the reward of policy gradient network when multiple states and actions exist per time step? I am working on a project, in which I'm using a policy gradient algorithm (REINFORCE) to select the best cleaning method/methods for erroneous samples in tabular datasets. The details are as follows. ... • 1 1 vote 0 answers 135 views In the policy gradient method, state dependent baseline does not affect gradient of the objective function. Then how this is better approach? In policy gradient theory subtracting state dependent baseline from Q(s,a) does not affect gradient of the objective function. I understand the proof shown below. One things is, if it's not affecting ... 0 votes 1 answer 167 views What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task? I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ... • 99 0 votes 1 answer 89 views How do I implement the 'gradient clipping' in the Neural Replicator Dynamics paper? The paper is here https://arxiv.org/pdf/1906.00190.pdf and the relevant paragraph where they explain their method is below: It's still not clear to me how this is meant to work exactly. In the pseudo-... 3 votes 1 answer 147 views Why might my policy gradient agent appear to maximize the absolute value of rewards? I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +... • 99 2 votes 0 answers 58 views For REINFORCE, why do different books give different algorithms? The discount rate appears twice in the REINFORCE algorithm in Sutton and Barto (2018). However, in three major books (Graesser and Keng (2020); Morales (2020); Ravichandiran (2020)) on reinforcement ... 2 votes 1 answer 745 views In the Policy Gradient Theorem proof, why is$d^\pi(s) = \sum_{k=0}^{\infty}\gamma^{k}Pr(s_0 \rightarrow s, k, \pi)\$ true?

I was reading the original Policy Gradient Paper. I didn't quiet get the last step of the proof for the policy gradient theorem. The proof given in the paper is below: I don't understand how the last ...
• 31