43 votes
Accepted

What is the relation between Q-learning and policy gradients methods?

However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG). Both ...
user avatar
  • 23.8k
20 votes
Accepted

How should I handle invalid actions (when using REINFORCE)?

Just ignore the invalid moves. For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
user avatar
14 votes

How should I handle invalid actions (when using REINFORCE)?

Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
user avatar
12 votes
Accepted

Why is the derivative of this objective function 0 if the policy is deterministic?

Here is the gradient that they are discussing in the video: $$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
user avatar
  • 9,379
12 votes

What is the relation between Q-learning and policy gradients methods?

This Tutorial by OpenAI offers a great comparison of different RL methods. I'll try to summarize the differences between Q-Learning and Policy Gradient methods: Objective Function In Q-Learning we ...
user avatar
11 votes
Accepted

Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
user avatar
  • 9,379
9 votes

Why are lambda returns so rarely used in policy gradients?

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
user avatar
9 votes
Accepted

Why does the "reward to go" trick in policy gradient methods work?

An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$): $$\mathbb{E}_{\tau \sim \pi_{\...
user avatar
  • 9,379
8 votes
Accepted

How can policy gradients be applied in the case of multiple continuous actions?

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
user avatar
7 votes
Accepted

Why does is make sense to normalize rewards per episode in reinforcement learning?

The "trick" of subtracting a (state-dependent) baseline from the $Q(s, a)$ term in policy gradients to reduce variants (which is what is described in your "baseline reduction" link) is a different ...
user avatar
  • 9,379
7 votes

Why is baseline conditional on state at some timestep unbiased?

Using the law of iterated expectations one has: $\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \...
user avatar
7 votes

How should I handle invalid actions (when using REINFORCE)?

I faced a similar issue recently with Minesweeper. The way I solved it was by ignoring the illegal/invalid moves entirely. Use the Q-network to predict the Q-values for all of your actions (valid ...
user avatar
  • 143
7 votes
Accepted

How is the policy gradient calculated in REINFORCE?

The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question. The gradient is ...
user avatar
6 votes

How should I handle invalid actions (when using REINFORCE)?

IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
user avatar
6 votes

How do I handle negative rewards in policy gradients with the cross-entropy loss function?

It depends on your loss function, but you probably need to tweak it. If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
user avatar
  • 161
5 votes

Do off-policy policy gradient methods exist?

Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions. The general idea of off-...
user avatar
5 votes

Why does is make sense to normalize rewards per episode in reinforcement learning?

This question is discussed in detail, in the following NeurIPS 2016 paper by David Silver: Learning values across many orders of magnitude. They also give experimental results over the Atari domain.
user avatar
5 votes
Accepted

Eligibility vector for softmax policy with policy gradients

Calculation of gradient \begin{align} \nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\ &= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a') \end{align} is only ...
user avatar
  • 2,226
5 votes
Accepted

What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?

The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA: The ...
user avatar
  • 23.8k
5 votes
Accepted

Why do the standard and deterministic Policy Gradient Theorems differ in their treatment of the derivatives of $R$ and the conditional probability?

In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the ...
user avatar
5 votes
Accepted

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor. The first is probably ...
user avatar
  • 876
4 votes
Accepted

Meaning of Actor Output in Actor Critic Reinforcement Learning

When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ...
user avatar
  • 9,379
4 votes
Accepted

What information should be cached in experience replay for actor-critic?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. ...
user avatar
4 votes

Why does is make sense to normalize rewards per episode in reinforcement learning?

We subtract mean from values and divide it with standard deviation to get data with mean of zero and variance of one. The range of values per episode does not matter, it will always make it to have ...
user avatar
  • 2,226
4 votes

What are the pros and cons of using standard deviation or entropy for exploration in PPO?

Both implementations may be closer than you think. In short: PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
user avatar
  • 141
4 votes
Accepted

How can the derivative of a neural network be calculated, given no mathematical expression?

I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate? Don't worry - lots of people get confused about this, because it seems ...
user avatar
  • 156
4 votes
Accepted

In the policy gradient equation, is $\pi(a_{t} | s_{t}, \theta)$ a distribution or a function?

First, the derivative is usually taken with respect to a variable (input) of the function. Hence the notation $\frac{df}{dx}$ for some function $f(x)$. If you look at your equation more carefully $$\...
user avatar
  • 33.8k
4 votes
Accepted

How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|...
user avatar
  • 2,226
4 votes
Accepted

What happens when you select actions using softmax instead of epsilon greedy in DQN?

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action. This is a very common choice, because it is simple to implement and quite robust....
user avatar
  • 23.8k

Only top scored, non community-wiki answers of a minimum length are eligible