Questions tagged [reinforce]

For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).

Filter by
Sorted by
Tagged with
2 votes
1 answer
127 views

REINFORCE with Baseline update rule

I was looking at the algorithm for REINFORCE with baseline from the Book 'Introduction to Reinforcement Learning' from Sutton: I do not quite understand the update rule for $w$: $w = w + \alpha \...
kklaw's user avatar
  • 175
2 votes
2 answers
49 views

Implementation difference in REINFORCE algorithm, where to sum from

I have a question regarding the implementation of the REINFORCE algorithm. In berkeley course (see slide 9) the gradient is defined as Note that the return sums from 1. However in Sutton's book the ...
Chris XU's user avatar
0 votes
1 answer
69 views

Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
Ufuk Can Bicici's user avatar
2 votes
1 answer
227 views

REINFORCE with Baseline not Learning

I have implemented REINFORCE using PyTorch and am testing it on the CartPole environment. My implementation allows for an optional baseline to be applied. At present, the baseline used is simply the ...
Beane's user avatar
  • 123
3 votes
1 answer
224 views

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
cfml's user avatar
  • 53
0 votes
1 answer
437 views

What is the problem in my implementation of actor critic?

I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1". As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
Labo's user avatar
  • 111
1 vote
0 answers
76 views

What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?

What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm? I know one possibility is to subtract a baseline as a running average of rewards from past ...
postnubilaphoebus's user avatar
2 votes
2 answers
483 views

Are batches useful for REINFORCE without strong episode cutoffs?

I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don'...
Josh's user avatar
  • 89
1 vote
1 answer
179 views

How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?

Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my ...
Josh's user avatar
  • 89
0 votes
1 answer
143 views

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
Josh's user avatar
  • 89
0 votes
1 answer
180 views

Should the concept of discounted rewards result in multiple arrays per episode in RL?

Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below: Say you have a reward array of length <...
Josh's user avatar
  • 89
3 votes
1 answer
141 views

Why might my policy gradient agent appear to maximize the absolute value of rewards?

I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
Josh's user avatar
  • 89
3 votes
1 answer
137 views

Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this: My ...
LarrySnyder610's user avatar
4 votes
0 answers
182 views

What is the correct interpretation of the discount factor in MDPs?

In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
Skander's user avatar
  • 41
1 vote
1 answer
564 views

How to implement PPO without using a Critic

I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
Aeryan's user avatar
  • 53
1 vote
0 answers
252 views

Why is an action-independent baseline required to reduce variance?

I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
Kronic's user avatar
  • 49
0 votes
0 answers
210 views

How to pass the rewards in zero-sum multiplayer context when using REINFORCE?

Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm. I have doubts about passing reward values in ...
hanugm's user avatar
  • 3,850
1 vote
1 answer
371 views

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

Consider the following algorithm from the textbook titled Reinforcement Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart While playing the game for the generation of ...
hanugm's user avatar
  • 3,850
1 vote
0 answers
223 views

FrozenLake-v0 not training using REINFORCE

I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all. I have used the same neural architecture ...
204's user avatar
  • 111
2 votes
0 answers
113 views

REINFORCE differentiation on sum or single value?

I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
Kronic's user avatar
  • 49
0 votes
0 answers
534 views

How to normalize rewards in REINFORCE?

I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
Mastiff's user avatar
  • 121
3 votes
1 answer
396 views

Why adding a baseline doesn't affect the policy gradient?

On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is an immediate consequence of the EGLP Lemma ...
Thomas Hustache's user avatar
1 vote
1 answer
184 views

How does the neural network learn when used in the REINFORCE algorithm?

As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good ...
desert_ranger's user avatar
1 vote
1 answer
343 views

Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function: $$ J\left(\...
Hao Huynh Nhat's user avatar
0 votes
1 answer
239 views

How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation: \begin{align} \nabla_{\theta}J &\propto \sum_s \mu(s) \...
Swakshar Deb's user avatar
2 votes
0 answers
33 views

What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?

Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
Marvis Lu's user avatar
  • 121
1 vote
1 answer
419 views

It is mathematically correct to use a Policy Gradient method for 1-step trajectories?

I have come across a Google paper that uses the REINFORCE algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be ...
Angelo's user avatar
  • 211
3 votes
0 answers
264 views

How to implement REINFORCE with eligibility traces?

The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
Javier Ventajas Hernández's user avatar
3 votes
2 answers
115 views

What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?

I am wondering what the parameter $y$ in the function $g(y,\mu,\sigma)=\frac{1}{(2\pi)^{1/2}\sigma}e^{-(y-\mu)^{2/2\sigma^2}}$ stands for in Section 6 (page 14) of the paper introducing the REINFORCE ...
Daniel B.'s user avatar
  • 815
1 vote
0 answers
524 views

Are actions deterministic during testing in continuous action space PPO?

In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
Mika's user avatar
  • 341
1 vote
0 answers
77 views

Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?

I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard. My ...
Kristof's user avatar
  • 26
2 votes
0 answers
331 views

Why is TD(0) not converging to the optimal policy?

I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy). Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
PeeteKeesel's user avatar
1 vote
0 answers
411 views

What adapts an algorithm to continuous or to discrete action spaces?

Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both REINFORCE and other policy ...
mugoh's user avatar
  • 531
1 vote
0 answers
197 views

If REINFORCE agent suddenly drops, how do I verify if it's due to catastrophic forgetting?

I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from ...
user3656142's user avatar
1 vote
0 answers
144 views

SeqGAN - Policy gradient objective function interpretation

Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here. Have I interpreted the ...
Mathav Raj's user avatar
8 votes
1 answer
1k views

Which loss function should I use in REINFORCE, and what are the labels?

I understand that this is the update for the parameters of a policy in REINFORCE: $$ \Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t}, $$ where $v_t$ is ...
S2673's user avatar
  • 590
4 votes
1 answer
198 views

Why does REINFORCE work at all?

Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book - I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why ...
stoic-santiago's user avatar
2 votes
0 answers
81 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
Igor's user avatar
  • 121
1 vote
1 answer
517 views

Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
pranav's user avatar
  • 191
3 votes
1 answer
2k views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
gnikol's user avatar
  • 175
2 votes
1 answer
234 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
junior-flight's user avatar
2 votes
0 answers
56 views

Can a typical supervised learning problem be solved with reinforcement learning methods?

Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning. I have a dataset of images and their matching ...
Gilad Deutsch's user avatar
3 votes
0 answers
230 views

Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function ...
GMV871's user avatar
  • 31
4 votes
1 answer
99 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
Habib-Allah's user avatar
1 vote
1 answer
140 views

Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?

I am interested in learning about policy gradient algorithms and REINFORCE. Can you suggest a good and easy paper that I can use to code them from scratch?
jgauth's user avatar
  • 261
5 votes
2 answers
2k views

How to calculate the advantage in policy gradient functions?

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
Bob Kimani's user avatar
2 votes
2 answers
371 views

What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
d56's user avatar
  • 233
3 votes
1 answer
274 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
BGa's user avatar
  • 229
1 vote
1 answer
503 views

How does the policy gradient's derivative work?

I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial. My first question is about the end result of this gradient derivation, \begin{aligned} \nabla \...
Eka's user avatar
  • 1,066
5 votes
1 answer
1k views

Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
Seewoo Lee's user avatar