Questions tagged [reinforce]
For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).
62
questions
0
votes
1
answer
48
views
Discount factors in REINFORCE Algorithm: Difference in two definitions
I am a bit confused between two definitions of the Vanilla REINFORCE algorithm.
The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/):
Here, at every step in an ...
- 101
-1
votes
0
answers
38
views
Why my Policy Gradient algo is minimizing the rewards instead of maximizing it?
I am using tensorflow 2.x to implement the REINFORCE algorithm for the Cartpole but instead of maximizing the rewards, it is minimizing the rewards. This algorithm is going to be implemented in my ...
1
vote
1
answer
47
views
REINFORCE with Baseline not Learning
I have implemented REINFORCE using PyTorch and am testing it on the CartPole environment. My implementation allows for an optional baseline to be applied. At present, the baseline used is simply the ...
- 113
2
votes
1
answer
150
views
$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)
I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition :
Generalize the box on page 199, the policy gradient theorem (13.5),
the proof of the policy ...
- 43
0
votes
1
answer
268
views
What is the problem in my implementation of actor critic?
I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1".
As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
- 111
0
votes
0
answers
24
views
Explain policy gradient proof from reinforcement learning book [duplicate]
How is this transition below happening? I don't understand.
Page 325 of rl book
http://www.incompleteideas.net/book/RLbook2020.pdf
- 101
1
vote
0
answers
65
views
What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?
What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?
I know one possibility is to subtract a baseline as a running average of rewards from past ...
2
votes
2
answers
282
views
Are batches useful for REINFORCE without strong episode cutoffs?
I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don'...
- 89
1
vote
1
answer
145
views
How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?
Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my ...
- 89
0
votes
1
answer
111
views
What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
- 89
0
votes
1
answer
132
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
- 89
3
votes
1
answer
91
views
Why might my policy gradient agent appear to maximize the absolute value of rewards?
I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
- 89
3
votes
1
answer
114
views
Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this:
My ...
- 133
4
votes
0
answers
132
views
What is the correct interpretation of the discount factor in MDPs?
In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
- 41
1
vote
1
answer
360
views
How to implement PPO without using a Critic
I am using the standard policy gradient algorithm, REINFORCE, to solve a RL problem and was thinking about implementing Proximal Policy Optimization (PPO) to increase the sample efficiency of my ...
- 53
1
vote
0
answers
225
views
Why is an action-independent baseline required to reduce variance?
I'm learning policy gradient methods. I encountered the REINFORCE algorithm with variance reduction with a baseline. I see we can use a constant or state-dependent function (e.g value function) but ...
- 49
0
votes
0
answers
182
views
How to pass the rewards in zero-sum multiplayer context when using REINFORCE?
Suppose there are two players in my zero-sum game and they play in a row like chess. And I want to learn the policy function using the REINFORCE algorithm.
I have doubts about passing reward values in ...
- 3,581
1
vote
1
answer
284
views
Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
Consider the following algorithm from the textbook titled Reinforcement
Learning: An Introduction (second edition) by Richard S. Sutton and Andrew G. Bart
While playing the game for the generation of ...
- 3,581
1
vote
0
answers
181
views
FrozenLake-v0 not training using REINFORCE
I am implementing a simple REINFORCE (policy gradient) algorithm for openAI's FrozenLake-v0 environment. However, it does not seem to learn anything at all.
I have used the same neural architecture ...
- 111
2
votes
0
answers
100
views
REINFORCE differentiation on sum or single value?
I'm currently learning Policy-gradient Methods for RL and encountered REINFORCE algorithm. I learned from this site : https://towardsdatascience.com/policy-gradient-methods-104c783251e0 that the ...
- 49
0
votes
0
answers
427
views
How to normalize rewards in REINFORCE?
I'm trying to solve a reinforcement learning problem using a Monte Carlo policy gradient algorithm and, more specifically, REINFORCE, with rewards attributed to individual moves instead of applied to ...
- 121
3
votes
1
answer
285
views
Why adding a baseline doesn't affect the policy gradient?
On the OpenAI's Spinning Up, they justify the fact that adding a baseline $b(s_t)$ in the policy gradient doesn't change its gradient by saying that this is
an immediate consequence of the EGLP Lemma
...
1
vote
1
answer
164
views
How does the neural network learn when used in the REINFORCE algorithm?
As per my understanding, you run an entire episode, which contains many steps, and then back-propagate using just a single loss value. How does the neural network learn to differentiate between good ...
- 588
1
vote
1
answer
295
views
Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?
I read the book "Foundation of Deep Reinforcement Learning, Laura Graesser and Wah Loon Keng", and when I go through the REINFORCE algorithm, they show the objective function:
$$
J\left(\...
0
votes
1
answer
199
views
How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:
\begin{align}
\nabla_{\theta}J &\propto \sum_s \mu(s) \...
- 673
2
votes
0
answers
32
views
What is the name of this algorithm that estimates the gradient with an average by sampling from a distribution?
Consider maximizing the function $R(w)$ with parameter $w$ using gradient ascent. However, we don't know the gradient $\nabla_wR(w)$ formula. Now suppose $w$ is sampled from a probability distribution ...
- 121
1
vote
1
answer
345
views
It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
I have come across a Google paper that uses the REINFORCE algorithm (a Policy Gradient Method) for a case where the trajectory of the episodes it proposes would be ...
- 201
3
votes
0
answers
225
views
How to implement REINFORCE with eligibility traces?
The pseudocode below is taken from Barto and Sutton's "Reinforcement Learning: an introduction". It shows an actor-critic implementation with eligibility traces. My question is: if I set $\...
3
votes
2
answers
104
views
What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?
I am wondering what the parameter $y$ in the function $g(y,\mu,\sigma)=\frac{1}{(2\pi)^{1/2}\sigma}e^{-(y-\mu)^{2/2\sigma^2}}$ stands for in Section 6 (page 14) of the paper introducing the REINFORCE ...
- 815
1
vote
0
answers
433
views
Are actions deterministic during testing in continuous action space PPO?
In a continuous action space (for instance, in PPO, TRPO, REINFORCE, etc.), during training, an action is sampled from the random distribution with $\mu$ and $\sigma$. This results in an inherent ...
- 331
1
vote
0
answers
67
views
Why do we use big batch/epoch size in policy gradient methods (vpg specifically)?
I am re-implementing vpg and using Spinning Up as reference implementation. I noticed that the default epoch size is 4000. I also see cues in papers that big batch size is quite standard.
My ...
- 26
2
votes
0
answers
260
views
Why is TD(0) not converging to the optimal policy?
I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy).
Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
- 121
1
vote
0
answers
370
views
What adapts an algorithm to continuous or to discrete action spaces?
Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both
REINFORCE and other policy ...
- 531
1
vote
0
answers
188
views
If REINFORCE agent suddenly drops, how do I verify if it's due to catastrophic forgetting?
I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from ...
- 185
1
vote
0
answers
140
views
SeqGAN - Policy gradient objective function interpretation
Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here.
Have I interpreted the ...
- 111
7
votes
1
answer
909
views
Which loss function should I use in REINFORCE, and what are the labels?
I understand that this is the update for the parameters of a policy in REINFORCE:
$$
\Delta \theta_{t}=\alpha \nabla_{\theta} \log \pi_{\theta}\left(a_{t} \mid s_{t}\right) v_{t},
$$
where $v_t$ is ...
- 560
4
votes
1
answer
182
views
Why does REINFORCE work at all?
Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book -
I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why ...
- 1,121
2
votes
0
answers
79
views
What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?
The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
- 121
1
vote
1
answer
405
views
Why is the "reward to go" replaced by Q instead of V, when transitioning from PG to actor critic methods?
While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5).
I am not ...
- 191
3
votes
1
answer
1k
views
Can I apply DQN or policy gradient algorithms in the contextual bandit setting?
I have a problem which I believe can be described as a contextual bandit.
More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
- 175
2
votes
1
answer
224
views
How long should the state-dependent baseline for policy gradient methods be trained at each iteration?
How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods?
I'm using this equation to ...
2
votes
0
answers
56
views
Can a typical supervised learning problem be solved with reinforcement learning methods?
Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning.
I have a dataset of images and their matching ...
- 629
3
votes
0
answers
199
views
Understanding the TensorFlow implementation of the policy gradient method
I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow.
I think I got almost everything. The only thing that still bothers me is the loss function ...
- 31
4
votes
1
answer
96
views
How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?
I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
- 51
1
vote
1
answer
115
views
Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?
I am interested in learning about policy gradient algorithms and REINFORCE. Can you suggest a good and easy paper that I can use to code them from scratch?
- 161
5
votes
2
answers
1k
views
How to calculate the advantage in policy gradient functions?
From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
- 153
2
votes
1
answer
292
views
What is the difference between Sutton's and Levine's REINFORCE algorithm?
I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below.
In particular, what does $i$ represent in the REINFORCE algorithm? ...
- 223
3
votes
1
answer
253
views
Purpose of using actor-critic algorithms under deterministic MDP dynamics?
One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
- 229
1
vote
1
answer
459
views
How does the policy gradient's derivative work?
I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial.
My first question is about the end result of this gradient derivation,
\begin{aligned}
\nabla \...
- 1,036
5
votes
1
answer
940
views
Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?
REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
- 173