# Questions tagged [reinforce]

For questions related to the REINFORCE algorithm (or update rule), which is a policy gradient algorithm, that is, an algorithm which estimates the policy directly (that is, without first estimating any value function).

62 questions
Filter by
Sorted by
Tagged with
48 views

### Discount factors in REINFORCE Algorithm: Difference in two definitions

I am a bit confused between two definitions of the Vanilla REINFORCE algorithm. The first one is in the following (from this page: https://stjohngrimbly.com/model-free-RL/): Here, at every step in an ...
38 views

### Why my Policy Gradient algo is minimizing the rewards instead of maximizing it?

I am using tensorflow 2.x to implement the REINFORCE algorithm for the Cartpole but instead of maximizing the rewards, it is minimizing the rewards. This algorithm is going to be implemented in my ...
1 vote
47 views

### REINFORCE with Baseline not Learning

I have implemented REINFORCE using PyTorch and am testing it on the CartPole environment. My implementation allows for an optional baseline to be applied. At present, the baseline used is simply the ...
150 views

### $\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
268 views

### What is the problem in my implementation of actor critic?

I have been implementing both REINFORCE with baseline and actor-critic to solve "cartpole-v1". As a reminder, here is the presentation of the algorithms in Sutton and Barto's book (http://...
24 views

### Explain policy gradient proof from reinforcement learning book [duplicate]

How is this transition below happening? I don't understand. Page 325 of rl book http://www.incompleteideas.net/book/RLbook2020.pdf
1 vote
65 views

### What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm?

What strategies are there to reduce the variance of the policy gradient estimator of the REINFORCE algorithm? I know one possibility is to subtract a baseline as a running average of rewards from past ...
282 views

### Are batches useful for REINFORCE without strong episode cutoffs?

I'm following along with PyTorch's example implementations (found here) of reinforcement learning algorithms that happen to be largely REINFORCE (vanilla policy gradient) based, and I notice they don'...
1 vote
145 views

### How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?

Say I have a simple multi-agent reinforcement learning problem using vanilla policy gradient methods (i.e. REINFORCE) that is currently running with one network per agent. If I can say that each of my ...
111 views

### What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

I am straying out of my domain knowledge to attempt a basic reinforcement learning task in a toy environment and have become fairly familiar with the REINFORCE algorithm for policy gradient agents, ...
132 views

### Should the concept of discounted rewards result in multiple arrays per episode in RL?

Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below: Say you have a reward array of length <...
91 views

### Why might my policy gradient agent appear to maximize the absolute value of rewards?

I have a toy policy gradient RL algorithm using REINFORCE (aka monte carlo policy gradients) that involves bots moving on a grid attempting to "acquire" targets in Pytorch. The bots receive +...
114 views

### Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this: My ...
132 views

253 views

### Purpose of using actor-critic algorithms under deterministic MDP dynamics?

One of the main disadvantages of the MC Policy Gradient algorithm (REINFORCE) as described say here is the fact that it has high variance (returns, which we sample, will significantly vary from ...
1 vote