22
votes
Accepted
How should I handle invalid actions (when using REINFORCE)?
Just ignore the invalid moves.
For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
- 4,295
14
votes
How should I handle invalid actions (when using REINFORCE)?
Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
- 3,857
11
votes
Accepted
Why are lambda returns so rarely used in policy gradients?
That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
- 10.5k
10
votes
Accepted
Why does the discount rate in the REINFORCE algorithm appear twice?
The discount factor does appear twice, and this is correct.
This is because the function you are trying to maximise in REINFORCE for an episodic problem (by taking the gradient) is the expected return ...
- 33.3k
9
votes
Why are lambda returns so rarely used in policy gradients?
Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
- 226
7
votes
Why does the discount rate in the REINFORCE algorithm appear twice?
Neil's answer already provides some intuition as to why the pseudocode (with the extra $\gamma^t$ term) is correct.
I'd just like to additionally clarify that you do not seem to be misunderstanding ...
- 10.5k
7
votes
Accepted
How is the policy gradient calculated in REINFORCE?
The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question.
The gradient is ...
- 2,064
7
votes
How should I handle invalid actions (when using REINFORCE)?
I faced a similar issue recently with Minesweeper.
The way I solved it was by ignoring the illegal/invalid moves entirely.
Use the Q-network to predict the Q-values for all of your actions (valid ...
- 163
6
votes
Accepted
Can I apply DQN or policy gradient algorithms in the contextual bandit setting?
MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance.
Therefore, methods used in RL to solve ...
- 33.3k
6
votes
How should I handle invalid actions (when using REINFORCE)?
IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
- 563
6
votes
Is REINFORCE the same as 'vanilla policy gradient'?
You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
- 498
5
votes
Accepted
Can we implement a memory in a REINFORCE algorithm for RL?
Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour.
...
- 33.3k
4
votes
Accepted
Confusion about temporal difference learning
My first question is whether the following "implementation" of the 𝑇𝐷(0) algorithm for the first two of the above observed trajectories correct?
$V(a)\leftarrow0 + 0.1(1+0-0)= 0.1; \quad ...
- 33.3k
4
votes
Accepted
Why does REINFORCE work at all?
The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$.
Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta)}...
- 5,030
4
votes
Accepted
Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the ...
- 156
4
votes
Accepted
What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?
One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the ...
3
votes
Accepted
Which loss function should I use in REINFORCE, and what are the labels?
The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
- 5,030
3
votes
Accepted
Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?
The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns, while in the pytorch version only the future is taken ...
- 171
3
votes
Why does the discount rate in the REINFORCE algorithm appear twice?
It's a subtle issue.
If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) ...
- 131
3
votes
Accepted
How does the neural network learn when used in the REINFORCE algorithm?
How does the neural network learn to differentiate between good and bad actions?
Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples ...
- 33.3k
3
votes
Accepted
Is it the high probability action that is always selected by the agent in REINFORCE algorithm?
You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
- 41.4k
3
votes
Accepted
REINFORCE with Baseline not Learning
Your baseline value is a mean of the returns of a single episode. This is correlated too much with the chosen actions for that episode.
Instead, use a baseline value which is a longer-running mean ...
- 33.3k
3
votes
Accepted
REINFORCE with Baseline update rule
The value $\delta$ is already representing a derivative equivalent to derivative of MSE loss for the difference between observed and predicted return. Multiplying it by the gradient of $\hat{v}$ to ...
- 33.3k
2
votes
How should I handle invalid actions (when using REINFORCE)?
An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
- 165
2
votes
Accepted
How to calculate the advantage in policy gradient functions?
First let us note the definition of the advantage function:
$$A(s,a) = Q(s,a) - V(s) \; ,$$
where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could ...
- 5,030
2
votes
How to calculate the advantage in policy gradient functions?
The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive ...
- 343
2
votes
Accepted
What is the difference between Sutton's and Levine's REINFORCE algorithm?
About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$.
...
- 395
2
votes
How is equation 8 derived in the paper "Self-critical sequence training for image captioning"?
First of all you made a mistake, equation 8 in the paper is defined with $\frac{\partial L(\theta)}{\partial s_t}$ not $\frac{\partial L(\theta)}{\partial\theta}$.
Loss is defined as:
$L(\theta) = -...
- 2,406
2
votes
Accepted
It is mathematically correct to use a Policy Gradient method for 1-step trajectories?
The fundamental idea behind policy gradient is just to maximise the return averaged across all probably trajectories, i.e
$$\begin{align} J(\theta) &= E[\sum\limits_{t=1}^{\tau}r(s_t,a_t)]\\
&=...
- 392
2
votes
Accepted
What does the parameter $y$ stand for in function $g(y,\mu,\sigma)$ related to REINFORCE algorithm?
If you take a look at the Wikipedia page related to the normal distribution, you will see the definition of the Gaussian density
$$
{\displaystyle f(x)={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {1}...
- 41.4k
Only top scored, non community-wiki answers of a minimum length are eligible