Questions tagged [reinforce]

For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).

Filter by
Sorted by
Tagged with
2
votes
0answers
26 views

Why is TD(0) not converging to the optimal policy?

I am trying to implement the basic RL algorithms to learn on this 10x10 GridWorld (from REINFORCEJS by Kaparthy). Currently I am stuck at TD(0). No matter how many episodes I run, when I am updating ...
1
vote
0answers
19 views

What adapts an algorithm to continuous or to discrete action spaces?

Some RL algorithms can only be used for environments with continuous action spaces (e.g TD3, SAC), while others only for discrete action spaces (DQN), and some for both REINFORCE and other policy ...
0
votes
0answers
14 views

REINFORCE Agent suddenly drops. How to verify if it's due to catastrophic forgetting?

I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from ...
1
vote
0answers
85 views

SeqGAN - Policy gradient objective function interpretation

Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here. Have I interpreted the ...
3
votes
1answer
111 views

Why does REINFORCE work at all?

Here's a screenshot of the popular policy-gradient algorithm from Sutton and Barto's book - I understand the mathematical derivation of the update rule - but I'm not able to build intuition as to why ...
1
vote
0answers
27 views

What is the proof that the variance of the gradient estimate in Actor-Critic is smaller than in REINFORCE?

The intuition provided when introducing actor-critic algorithms is that the variance of its gradient estimates is smaller than in REINFORCE as, e.g., discussed here. This intuition makes sense for the ...
0
votes
0answers
24 views

How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a ...
0
votes
0answers
30 views

Patient PPO: how to handle imbalanced discrete action space?

PPO agent. The action space includes 3 actions: 0: do nothing 1: act (start) 2: stop The agent has to perform thousands of steps doing nothing, then perform step 1 only once (act), then do nothing ...
1
vote
1answer
47 views

Why is the “reward to go” replaced by Q instead of V, when transitioning from PG to actor critic methods?

While transitioning from simple policy gradient to the actor-critic algorithm, most sources begin by replacing the "reward to go" with the state-action value function (see this slide 5). I am not ...
1
vote
1answer
207 views

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

I have a problem which I believe can be described as a contextual bandit. More specifically, in each round, I observe a context from the environment consisting of five continuous features, and, ...
2
votes
1answer
109 views

How long should the state-dependent baseline for policy gradient methods be trained at each iteration?

How long should the state-dependent baseline be trained at each iteration? Or what baseline loss should we target at each iteration for use with policy gradient methods? I'm using this equation to ...
2
votes
0answers
48 views

Can a typical supervised learning problem be solved with reinforcement learning methods?

Let's say I want to teach a neural to classify images, and, for some reason, I insist on using reinforcement learning rather than supervised learning. I have a dataset of images and their matching ...
3
votes
0answers
55 views

Understanding the TensorFlow implementation of the policy gradient method

I was trying to understand the implementation of a basic policy gradient (REINFORCE) method using TensorFlow. I think I got almost everything. The only thing that still bothers me is the loss function ...
2
votes
1answer
53 views

How can I sample the output distribution multiple times when pruning the filters with reinforcement learning?

I was reading the paper Learning to Prune Filters in Convolutional Neural Networks, which is about pruning the CNN filters using reinforcement learning (policy gradient). The paper says that the input ...
1
vote
1answer
41 views

Is there a good and easy paper to code policy gradient algorithms (REINFORCE) from scratch?

I am interested in learning about policy gradient algorithms and REINFORCE. Can you suggest a good and easy paper that I can use to code them from scratch?
4
votes
2answers
115 views

How to calculate the advantage in policy gradient functions?

From my understanding of the REINFORCE policy gradient method, we gently nudge the probabilities of actions based on the advantages. More specifically, the positive advantages increase the ...
2
votes
1answer
100 views

What is the difference between Sutton's and Levine's REINFORCE algorithm?

I followed the videos/slides of Berkley RL course, but now I am a bit confused when implementing it. Please see the picture below. In particular, what does $i$ represent in the REINFORCE algorithm? ...
3
votes
1answer
74 views

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
1
vote
1answer
112 views

How does the policy gradient's derivative work?

I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial. My first question is about the end result of this gradient derivation, \begin{aligned} \nabla \...
4
votes
1answer
183 views

Update in REINFORCE algorithm - step-wise or episode-wise?

REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
4
votes
1answer
72 views

REINFORCE algorithm for portfolio optimization - problem while training

I'm trying to implement the Reinforce algorithm (Monte Carlo policy gradient) in order to optimize a portfolio of 94 stocks on a daily basis (I have suitable historical data to achieve this). The idea ...
3
votes
1answer
54 views

Confusion about temporal difference learning

I have a couple of small questions about the David Silver lecture about reinforcement learning, lecture slides (slides 23, 24). More specifically it is about the temporal difference algorithm: $$V(...
4
votes
0answers
261 views

Policy gradient methods for continuous action space

I have a problem I would like to tackle with RL but I am not sure if it is even doable. My agent has to figure out how to fill a very large vector (let's say from 600 to 4000 in the most complex ...
1
vote
0answers
35 views

How is computed the gradient with respect to each output node from a loss value?

newbie here. I am studying the REINFORCE method in "Deep Reinforcement Learning Hands-On". I can't understand how, after computing the loss of the episode, that loss is backpropagated in a NN with ...
3
votes
1answer
123 views

How is REINFORCE used instead of Backpropagation?

In neural networks with stochastic layers I've seen the use of the REINFORCE estimator for estimating the gradient (because it can't be computed directly). Some such examples are Show, Attend and ...
6
votes
1answer
405 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
0answers
193 views

Policy gradient loss for neural network training

Say i want to train a neural network with 10 classes as outputs and use categorical_cross_entropy as a loss function in keras. This will try to fit the training ...
1
vote
1answer
1k views

What is the use of the seed function in the gym Environment 'Pendulum-v0'?

Why do we use the seed function in the 'Pendulum-v0' environment? https://github.com/openai/gym/blob/master/gym/envs/classic_control/pendulum.py#L25
3
votes
1answer
138 views

How is equation 8 derived in the paper “Self-critical sequence training for image captioning”?

In the paper "Self-critical sequence training for image captioning", on page 3, they define the loss function (of the parameters $\theta$) of an image captioning system as the negative expected reward ...
5
votes
2answers
614 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
14
votes
3answers
1k views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...
28
votes
4answers
9k views

How should I handle invalid actions (when using REINFORCE)?

I want to create an AI which can play five-in-a-row/gomoku. I want to use reinforcement learning for this. I use policy gradient method, namely REINFORCE, with baseline. For the value and policy ...