Questions tagged [reinforce]

For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).

Filter by
Sorted by
Tagged with
1
vote
0answers
26 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
0
votes
0answers
23 views

How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
28 views

Patient PPO: how to handle imbalanced discrete action space?

PPO agent. The action space includes 3 actions: 0: do nothing 1: act (start) 2: stop The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
1
vote
1answer
40 views

Why is the “reward to go” replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
1
vote
1answer
134 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
2
votes
1answer
101 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
2
votes
0answers
46 views

Can a typical supervised learning problem be solved with reinforcement learning methods?

Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning. I have a dataset of images and their matching ...
3
votes
0answers
50 views

Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function ...
2
votes
1answer
48 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
1
vote
1answer
39 views

Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?

I am interested in learning about policy gradient algorithms and REINFORCE. Can you suggest a good and easy paper that I can use to code them from scratch?
3
votes
2answers
91 views

How to calculate the advantage in policy gradient functions?

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
2
votes
1answer
85 views

What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
3
votes
1answer
65 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
1
vote
1answer
85 views

How does the policy gradient's derivative work?

I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial. My first question is about the end result of this gradient derivation, \begin{aligned} \nabla \...
2
votes
1answer
126 views

Update in REINFORCE algorithm - step-wise or episode-wise?

REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
4
votes
1answer
66 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
3
votes
1answer
45 views

Confusion about temporal difference learning

I have a couple of small questions about the David Silver lecture about reinforcement learning, lecture slides (slides 23, 24). More specifically it is about the temporal difference algorithm: $$V(...
3
votes
0answers
231 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
1
vote
0answers
33 views

How is computed the gradient with respect to each output node from a loss value?

newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with ...
3
votes
1answer
103 views

How is REINFORCE used instead of Backpropagation?

In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly). Some such examples are Show, Attend and ...
4
votes
1answer
329 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
0answers
171 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
1
vote
1answer
776 views

What is the use of the seed function in the gym Environment 'Pendulum-v0'?

Why do we use the seed function in the 'Pendulum-v0' environment? https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L25
3
votes
1answer
130 views

How is equation 8 derived in the paper “Self-critical sequence training for image captioning”?

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward ...
8
votes
3answers
3k views

Why does is make sense to normalize rewards per episode in reinforcement learning?

In Open AI's actor-critic and in Open AI's REINFORCE, the rewards are being normalized like so rewards = (rewards - rewards.mean()) / (rewards.std() + eps) on ...
5
votes
2answers
501 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
13
votes
3answers
1k views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...