Questions tagged [reinforce]

For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).

Filter by
Sorted by
Tagged with
2
votes
0answers
34 views

How to calculate the advantage in policy gradient functions?

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
2
votes
1answer
77 views

What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
3
votes
1answer
47 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
1
vote
1answer
62 views

How is the policy gradient's derivative work?

I am trying to understand policy gradient method using a pytorch implementation and this tutorial. My first question is about the end result of this gradient derivation, mainly in this equation $\...
2
votes
1answer
102 views

Update in REINFORCE algorithm - step-wise or episode-wise?

REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
2
votes
1answer
50 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
3
votes
1answer
41 views

Confusion about temporal difference learning

I have a couple of small questions about the David Silver lecture about reinforcement learning, lecture slides (slides 23, 24). More specifically it is about the temporal difference algorithm: $$V(...
3
votes
0answers
169 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
1
vote
0answers
31 views

How is computed the gradient with respect to each output node from a loss value?

newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with ...
3
votes
1answer
74 views

How is REINFORCE used instead of Backpropagation?

In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly). Some such examples are Show, Attend and ...
4
votes
1answer
238 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
0answers
143 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
0
votes
1answer
409 views

What is the use of the seed function in the gym Environment 'Pendulum-v0'?

Why do we use the seed function in the 'Pendulum-v0' environment? https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L25
3
votes
1answer
109 views

How is equation 8 derived in the paper “Self-critical sequence training for image captioning”?

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward ...
8
votes
3answers
2k views

Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
11
votes
3answers
904 views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...