# Tag Info

## Hot answers tagged reinforce

Accepted

### How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
• 4,077

### How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
• 3,627
Accepted

• 226
Accepted

### How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ...
• 1,948

### How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
• 143
Accepted

### Can I apply DQN or policy gradient algorithms in the contextual bandit setting?

MDPs are strict generalisations of contextual bandits, adding time steps and state transitions, plus the concept of return as a measure of agent performance. Therefore, methods used in RL to solve ...
• 24.6k

### How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
• 515

### Why does the discount rate in the REINFORCE algorithm appear twice?

Neil's answer already provides some intuition as to why the pseudocode (with the extra $\gamma^t$ term) is correct. I'd just like to additionally clarify that you do not seem to be misunderstanding ...
• 9,509
Accepted

• 4,198
Accepted

### Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

I'm actually working on this example too, implemented the REINFORCE algorithm, and got the same result as you. My only guess is that the authors chose a different initial $\theta$ value to show the ...
• 56
Accepted

### Which loss function should I use in REINFORCE, and what are the labels?

The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
• 4,198

### Is REINFORCE the same as 'vanilla policy gradient'?

You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
• 93

### Why does the discount rate in the REINFORCE algorithm appear twice?

It's a subtle issue. If you look at the A3C algorithm in the original paper (p.4 and appendix S3 for pseudo-code), their actor-critic algorithm (same algorithm both episodic and continuing problems) ...
• 131
Accepted

### How does the neural network learn when used in the REINFORCE algorithm?

How does the neural network learn to differentiate between good and bad actions? Good actions - in context of a given state - have higher return than bad actions on average, taken over many examples ...
• 24.6k
Accepted

### What modifications can maximize the efficacy of the REINFORCE algorithm for a policy gradient task?

One simple improvement over the REINFORCE algorithm you've linked to is to use the advantage function instead of the normalised cumulative discounted return. The implementation is can be found in the ...

### How should I handle invalid actions (when using REINFORCE)?

An experimental paper exist in arxiv about the effect of whether to mask or to give negative rewards to invalid actions. There are some references in this paper which also discuss the effects and the ...
• 143
Accepted

### How does the policy gradient's derivative work?

You cannot do this: $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$ That ...
• 24.6k
Accepted

### Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?

The essence of your observation is that Sutton's version of REINFORCE is taking into consideration all of the trajectory to compute the returns, while in the pytorch version only the future is taken ...
Accepted

### What is the difference between Sutton's and Levine's REINFORCE algorithm?

About the first question, you are right. The $i$ denotes a sample trajectory corresponding to a whole episode. However, Sutton's version is exactly the same one as Levine's if you choose $N=1$. ...
• 395
Accepted

### How to calculate the advantage in policy gradient functions?

First let us note the definition of the advantage function: $$A(s,a) = Q(s,a) - V(s) \; ,$$ where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could ...
• 4,198
Accepted

• 4,198
Accepted

### Is it the high probability action that is always selected by the agent in REINFORCE algorithm?

You sample according to the probability distribution $\pi(a \mid s, \theta)$, so you do not always take the action with the highest probability (otherwise there would be no exploration but just ...
• 34.9k
Accepted

### Should the concept of discounted rewards result in multiple arrays per episode in RL?

The discount factor is not used to take into account uncertainty, but to encourage the agent to have a longer term view when it takes an action. If the discount factor is close to 0, the agent is ...
Accepted

### How can rewards and loss calculation be extended to multiple agents in a vanilla policy gradient RL setting?

Yes, this can be done and is widely applied in recent literature on multi-agent RL, at least with the collaborative setting where agents are optimizing a shared reward. This is also known as parameter ...
• 438
1 vote
Accepted

### Why does the implementation of REINFORCE algorithm minimize the gradient term but not the loss?

They are not maximizing the gradient, the gradient is of the form \begin{equation} \nabla_{\theta} J \approx \sum_{t=0}^T G_t \nabla_{\theta} \log(\pi_{\theta}(a_t|s_t)) \end{equation} that means that ...
• 2,236

Only top scored, non community-wiki answers of a minimum length are eligible