# Tag Info

Accepted

### How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...

### How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
Accepted

### Why does the discount rate in the REINFORCE algorithm appear twice?

Neil's answer already provides some intuition as to why the pseudocode (with the extra $\gamma^t$ term) is correct. I'd just like to additionally clarify that you do not seem to be misunderstanding ...
Accepted

### How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ...

### How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
Accepted

### Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...

### How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...

### Is REINFORCE the same as 'vanilla policy gradient'?

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
Accepted

Accepted

### Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the ...
Accepted

### What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the ...
Accepted

### Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
Accepted

### Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns, while in the pytorch version only the future is taken ...

### Why does the discount rate in the REINFORCE algorithm appear twice?

It's a subtle issue. If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) ...
Accepted

### How does the neural network learn when used in the REINFORCE algorithm?

How does the neural network learn to differentiate between good and bad actions? Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples ...
Accepted

### Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
Accepted

### REINFORCE with Baseline not Learning

Your baseline value is a mean of the returns of a single episode. This is correlated too much with the chosen actions for that episode. Instead, use a baseline value which is a longer-running mean ...

### How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
Accepted

First let us note the definition of the advantage function: $$A(s,a) = Q(s,a) - V(s) \; ,$$ where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could ...

The advantage is basically a function of the actual return received and a baseline. The function of the baseline is to make sure that only the actions that are better than average receive a positive ...
About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$. ...