21 votes
Accepted

How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
14 votes

How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
11 votes
Accepted

Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
  • 9,509
9 votes
Accepted

Why does the discount rate in the REINFORCE algorithm appear twice?

The discount factor does appear twice, and this is correct. This is because the function you are trying to maximise in REINFORCE for an episodic problem (by taking the gradient) is the expected return ...
  • 24.6k
9 votes

Why are lambda returns so rarely used in policy gradients?

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
7 votes
Accepted

How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ...
7 votes

How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
  • 143
6 votes
Accepted

Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
  • 24.6k
6 votes

How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
6 votes

Why does the discount rate in the REINFORCE algorithm appear twice?

Neil's answer already provides some intuition as to why the pseudocode (with the extra $\gamma^t$ term) is correct. I'd just like to additionally clarify that you do not seem to be misunderstanding ...
  • 9,509
4 votes
Accepted

Confusion about temporal difference learning

My first question is whether the following "implementation" of the 𝑇𝐷(0) algorithm for the first two of the above observed trajectories correct? $V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad ...
  • 24.6k
4 votes
Accepted

Why does REINFORCE work at all?

The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$. Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}...
4 votes
Accepted

Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the ...
  • 56
3 votes
Accepted

Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
3 votes

Is REINFORCE the same as 'vanilla policy gradient'?

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
  • 93
3 votes

Why does the discount rate in the REINFORCE algorithm appear twice?

It's a subtle issue. If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) ...
  • 131
3 votes
Accepted

How does the neural network learn when used in the REINFORCE algorithm?

How does the neural network learn to differentiate between good and bad actions? Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples ...
  • 24.6k
3 votes
Accepted

What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the ...
2 votes

How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
  • 143
2 votes
Accepted

How does the policy gradient's derivative work?

You cannot do this: $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$ That ...
  • 24.6k
2 votes
Accepted

Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns, while in the pytorch version only the future is taken ...
2 votes
Accepted

What is the difference between Sutton's and Levine's REINFORCE algorithm?

About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$. ...
2 votes
Accepted

How to calculate the advantage in policy gradient functions?

First let us note the definition of the advantage function: $$A(s,a) = Q(s,a) - V(s) \; ,$$ where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could ...
2 votes
Accepted

What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?

If you take a look at the Wikipedia page related to the normal distribution, you will see the definition of the Gaussian density $$ {\displaystyle f(x)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}...
  • 34.9k
2 votes

How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?

First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$. Loss is defined as: $L(\theta) = -...
  • 2,236
2 votes

Why adding a baseline doesn't affect the policy gradient?

The policy gradient states that $$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$ where the derivatives are taken wrt the parameter $\theta$. Now, if we say ...
2 votes
Accepted

Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
  • 34.9k
2 votes
Accepted

Should the concept of discounted rewards result in multiple arrays per episode in RL?

The discount factor is not used to take into account uncertainty, but to encourage the agent to have a longer term view when it takes an action. If the discount factor is close to 0, the agent is ...
2 votes
Accepted

How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?

Yes, this can be done and is widely applied in recent literature on multi-agent RL, at least with the collaborative setting where agents are optimizing a shared reward. This is also known as parameter ...
  • 438
1 vote
Accepted

Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

They are not maximizing the gradient, the gradient is of the form \begin{equation} \nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t)) \end{equation} that means that ...
  • 2,236

Only top scored, non community-wiki answers of a minimum length are eligible