Share Your Experience: Take the 2024 Developer Survey

# Tag Info

Accepted

### What is the relation between Q-learning and policy gradients methods?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Both ...
• 32.5k
Accepted

### How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
• 4,275

### What is the relation between Q-learning and policy gradients methods?

This Tutorial by OpenAI offers a great comparison of different RL methods. I'll try to summarize the differences between Q-Learning and Policy Gradient methods: Objective Function In Q-Learning we ...
• 451

### How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
• 3,827
Accepted

• 10.3k
Accepted

• 226
Accepted

### Why does is make sense to normalize rewards per episode in reinforcement learning?

The "trick" of subtracting a (state-dependent) baseline from the $Q(s, a)$ term in policy gradients to reduce variants (which is what is described in your "baseline reduction" link) is a different ...
• 10.3k
Accepted

### How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
• 3,827
Accepted

### Are policy gradient methods good for large discrete action spaces?

I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous ...
• 2,534

Using the law of iterated expectations one has: $\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \... 7 votes ### How do I handle negative rewards in policy gradients with the cross-entropy loss function? It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ... • 171 7 votes ### How should I handle invalid actions (when using REINFORCE)? I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ... • 163 7 votes Accepted ### How is the policy gradient calculated in REINFORCE? The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ... • 2,056 7 votes Accepted ### What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic? The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA: The ... • 32.5k 6 votes ### Do off-policy policy gradient methods exist? Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions. The general idea of off-... • 3,827 6 votes ### How should I handle invalid actions (when using REINFORCE)? IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ... • 553 6 votes ### Why does is make sense to normalize rewards per episode in reinforcement learning? This question is discussed in detail, in the following NeurIPS 2016 paper by David Silver: Learning values across many orders of magnitude. They also give experimental results over the Atari domain. 6 votes ### Is REINFORCE the same as 'vanilla policy gradient'? You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ... • 498 6 votes Accepted ### Why do the standard and deterministic Policy Gradient Theorems differ in their treatment of the derivatives of$R$and the conditional probability? In the policy gradient theorem, we don't need to write$r$as a function of$a$because the only time we explicitly 'see'$ris when we are taking the expectation with respect to the policy. For the ... • 4,920 5 votes Accepted ### Meaning of Actor Output in Actor Critic Reinforcement Learning When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ... • 10.3k 5 votes Accepted ### How can the derivative of a neural network be calculated, given no mathematical expression? I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems ... • 166 5 votes Accepted ### Eligibility vector for softmax policy with policy gradients Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ... • 2,406 5 votes Accepted ### How does being on-policy prevent us from using the replay buffer with the policy gradients? Let's say your old policy is\pi_b$and your current one is$\pi_a$. If you collected trajectory by using policy$\pi_b$you would get return$Gwhose expected value is \begin{align} E_{\pi_b}[G_t|... • 2,406 5 votes Accepted ### If\gamma \in (0,1)\$, what is the on-policy state distribution for episodic tasks?

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor. The first is probably ...
• 1,261

### Why does is make sense to normalize rewards per episode in reinforcement learning?

We subtract mean from values and divide it with standard deviation to get data with mean of zero and variance of one. The range of values per episode does not matter, it will always make it to have ...
• 2,406
Accepted

### What information should be cached in experience replay for actor-critic?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. ...
• 156