46
votes
Accepted
What is the relation between Q-learning and policy gradients methods?
However, both approaches appear identical to me i.e. predicting the maximum reward for an action (Q-learning) is equivalent to predicting the probability of taking the action directly (PG).
Both ...
- 30.2k
21
votes
Accepted
How should I handle invalid actions (when using REINFORCE)?
Just ignore the invalid moves.
For exploration, it is likely that you won't just execute the move with the highest probability, but instead choose moves randomly based on the outputted probability. If ...
- 4,205
14
votes
What is the relation between Q-learning and policy gradients methods?
This Tutorial by OpenAI offers a great comparison of different RL methods.
I'll try to summarize the differences between Q-Learning and Policy Gradient methods:
Objective Function
In Q-Learning we ...
- 451
14
votes
How should I handle invalid actions (when using REINFORCE)?
Usually softmax methods in policy gradient methods using linear function approximation use the following formula to calculate the probability of choosing action $a$. Here, weights are $\theta$, and ...
- 3,787
13
votes
Accepted
Why is the derivative of this objective function 0 if the policy is deterministic?
Here is the gradient that they are discussing in the video:
$$\nabla_{\theta} J(\theta) \approx \frac{1}{N} \sum_{i=1}^N \left( \sum_{t=1}^T \nabla_{\theta} \log \pi_{\theta} (\mathbf{a}_{i, t} \vert \...
- 10.1k
11
votes
Accepted
Why are lambda returns so rarely used in policy gradients?
That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
- 10.1k
10
votes
Accepted
Why does the "reward to go" trick in policy gradient methods work?
An important thing we're going to need is what is called the "Expected Grad-Log-Prob Lemma here" (proof included on that page), which says that (for any $t$):
$$\mathbb{E}_{\tau \sim \pi_{\...
- 10.1k
9
votes
Why are lambda returns so rarely used in policy gradients?
Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
- 226
8
votes
Accepted
Why does is make sense to normalize rewards per episode in reinforcement learning?
The "trick" of subtracting a (state-dependent) baseline from the $Q(s, a)$ term in policy gradients to reduce variants (which is what is described in your "baseline reduction" link) is a different ...
- 10.1k
8
votes
Accepted
How can policy gradients be applied in the case of multiple continuous actions?
As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. ...
- 3,787
8
votes
Accepted
Are policy gradient methods good for large discrete action spaces?
I don't think that (at least from a practical standpoint), there is much difference between continuous action space and discrete action space with >2k discrete actions. Quoting the "Continuous ...
- 2,426
7
votes
Why is baseline conditional on state at some timestep unbiased?
Using the law of iterated expectations one has:
$\triangledown _\theta \sum_{t=1}^T \mathbb{E}_{(s_t,a_t) \sim p(s_t,a_t)} [b(s_t)] = \nabla_\theta \sum_{t=1}^T \mathbb{E}_{s_t \sim p(s_t)} \left[ \...
- 171
7
votes
How do I handle negative rewards in policy gradients with the cross-entropy loss function?
It depends on your loss function, but you probably need to tweak it.
If you are using an update rule like loss = -log(probabilities) * reward, then your loss is ...
- 171
7
votes
How should I handle invalid actions (when using REINFORCE)?
I faced a similar issue recently with Minesweeper.
The way I solved it was by ignoring the illegal/invalid moves entirely.
Use the Q-network to predict the Q-values for all of your actions (valid ...
- 163
7
votes
Accepted
How is the policy gradient calculated in REINFORCE?
The first part of this answer is a little background that might bolster your intuition for what's going on. The second part is the more practical and direct answer to your question.
The gradient is ...
- 2,028
7
votes
Accepted
What is the difference between vanilla policy gradient with a baseline as value function and advantage actor-critic?
The difference between Vanilla Policy Gradient (VPG) with a baseline as value function and Advantage Actor-Critic (A2C) is very similar to the difference between Monte Carlo Control and SARSA:
The ...
- 30.2k
6
votes
Do off-policy policy gradient methods exist?
Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions.
The general idea of off-...
- 3,787
6
votes
How should I handle invalid actions (when using REINFORCE)?
IMHO the idea of invalid moves is itself invalid. Imagine placing an "X" at coordinates (9, 9). You could consider it to be an invalid move and give it a negative ...
- 553
6
votes
Why does is make sense to normalize rewards per episode in reinforcement learning?
This question is discussed in detail, in the following NeurIPS 2016 paper by David Silver: Learning values across many orders of magnitude. They also give experimental results over the Atari domain.
- 61
6
votes
Accepted
Why do the standard and deterministic Policy Gradient Theorems differ in their treatment of the derivatives of $R$ and the conditional probability?
In the policy gradient theorem, we don't need to write $r$ as a function of $a$ because the only time we explicitly 'see' $r$ is when we are taking the expectation with respect to the policy. For the ...
- 4,665
5
votes
Accepted
Meaning of Actor Output in Actor Critic Reinforcement Learning
When using the loss function for the critic described in your question, the Actor-Critic is an on-policy approach (as are most Actor-Critic methods). Your intuition as to what it is learning seems to ...
- 10.1k
5
votes
Accepted
How can the derivative of a neural network be calculated, given no mathematical expression?
I think what you mean to ask is how can differentiation occur when there's no obvious neural network function to differentiate?
Don't worry - lots of people get confused about this, because it seems ...
- 166
5
votes
Accepted
Eligibility vector for softmax policy with policy gradients
Calculation of gradient
\begin{align}
\nabla_{\theta} \log(\pi_{\theta}(a|s)) &= \phi(s,a) - \mathbb E[\phi (s, \cdot)]\\
&= \phi(s,a) - \sum_{a'} \pi(a'|s) \phi(s,a')
\end{align}
is only ...
- 2,336
5
votes
Accepted
How does being on-policy prevent us from using the replay buffer with the policy gradients?
Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is
\begin{align}
E_{\pi_b}[G_t|...
- 2,336
5
votes
Accepted
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor.
The first is probably ...
- 1,201
4
votes
Why does is make sense to normalize rewards per episode in reinforcement learning?
We subtract mean from values and divide it with standard deviation to get data with mean of zero and variance of one. The range of values per episode does not matter, it will always make it to have ...
- 2,336
4
votes
Accepted
What information should be cached in experience replay for actor-critic?
The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. ...
- 156
4
votes
Is REINFORCE the same as 'vanilla policy gradient'?
You can check the Open AI Introduction to RL series, they explain pretty neatly there what is the Policy Optimization and how to derive it. I think, that usually when we are talking about REINFORCE ...
- 113
4
votes
What are the pros and cons of using standard deviation or entropy for exploration in PPO?
Both implementations may be closer than you think.
In short:
PPO has both parts: there is noisiness in draws during training (with learned standard deviation), helping to explore new promising ...
- 141
Only top scored, non community-wiki answers of a minimum length are eligible