Skip to main content
22 votes
Accepted

How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
BlindKungFuMaster's user avatar
14 votes

How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
Jaden Travnik's user avatar
11 votes
Accepted

Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
Dennis Soemers's user avatar
  • 10.4k
10 votes
Accepted

Why does the discount rate in the REINFORCE algorithm appear twice?

The discount factor does appear twice, and this is correct. This is because the function you are trying to maximise in REINFORCE for an episodic problem (by taking the gradient) is the expected return ...
Neil Slater's user avatar
  • 32.7k
9 votes

Why are lambda returns so rarely used in policy gradients?

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
Brett Daley's user avatar
7 votes

Why does the discount rate in the REINFORCE algorithm appear twice?

Neil's answer already provides some intuition as to why the pseudocode (with the extra $\gamma^t$ term) is correct. I'd just like to additionally clarify that you do not seem to be misunderstanding ...
Dennis Soemers's user avatar
  • 10.4k
7 votes
Accepted

How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ...
Philip Raeisghasem's user avatar
7 votes

How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
Sanavesa's user avatar
  • 163
6 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
Neil Slater's user avatar
  • 32.7k
6 votes

How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
maaartinus's user avatar
6 votes

Is REINFORCE the same as 'vanilla policy gradient'?

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
vl_knd's user avatar
  • 498
5 votes
Accepted

Can we implement a memory in a REINFORCE algorithm for RL?

Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour. ...
Neil Slater's user avatar
  • 32.7k
4 votes
Accepted

Confusion about temporal difference learning

My first question is whether the following "implementation" of the 𝑇𝐷(0) algorithm for the first two of the above observed trajectories correct? $V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad ...
Neil Slater's user avatar
  • 32.7k
4 votes
Accepted

Why does REINFORCE work at all?

The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$. Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}...
David's user avatar
  • 4,920
4 votes
Accepted

Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the ...
gwtw14's user avatar
  • 156
4 votes
Accepted

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the ...
Raphael Lopez Kaufman's user avatar
3 votes
Accepted

Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
David's user avatar
  • 4,920
3 votes
Accepted

Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns, while in the pytorch version only the future is taken ...
Dimitris Monroe's user avatar
3 votes

Why does the discount rate in the REINFORCE algorithm appear twice?

It's a subtle issue. If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) ...
toto2's user avatar
  • 131
3 votes
Accepted

How does the neural network learn when used in the REINFORCE algorithm?

How does the neural network learn to differentiate between good and bad actions? Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
nbro's user avatar
  • 40.9k
3 votes
Accepted

REINFORCE with Baseline not Learning

Your baseline value is a mean of the returns of a single episode. This is correlated too much with the chosen actions for that episode. Instead, use a baseline value which is a longer-running mean ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

REINFORCE with Baseline update rule

The value $\delta$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to ...
Neil Slater's user avatar
  • 32.7k
2 votes

How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
Sanyou's user avatar
  • 165
2 votes
Accepted

How to calculate the advantage in policy gradient functions?

First let us note the definition of the advantage function: $$A(s,a) = Q(s,a) - V(s) \; ,$$ where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could ...
David's user avatar
  • 4,920
2 votes

How to calculate the advantage in policy gradient functions?

The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive ...
pecey's user avatar
  • 333
2 votes
Accepted

What is the difference between Sutton's and Levine's REINFORCE algorithm?

About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$. ...
Diego Gomez's user avatar
2 votes

How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?

First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$. Loss is defined as: $L(\theta) = -...
Brale's user avatar
  • 2,406
2 votes
Accepted

It is mathematically correct to use a Policy Gradient method for 1-step trajectories?

The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e $$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\ &=...
quest ions's user avatar
2 votes
Accepted

What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?

If you take a look at the Wikipedia page related to the normal distribution, you will see the definition of the Gaussian density $$ {\displaystyle f(x)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}...
nbro's user avatar
  • 40.9k

Only top scored, non community-wiki answers of a minimum length are eligible