Search Results
| Search type | Search syntax |
|---|---|
| Tags | [tag] |
| Exact | "words here" |
| Author |
user:1234 user:me (yours) |
| Score |
score:3 (3+) score:0 (none) |
| Answers |
answers:3 (3+) answers:0 (none) isaccepted:yes hasaccepted:no inquestion:1234 |
| Views | views:250 |
| Code | code:"if (foo != bar)" |
| Sections |
title:apples body:"apples oranges" |
| URL | url:"*.example.com" |
| Saves | in:saves |
| Status |
closed:yes duplicate:no migrated:no wiki:no |
| Types |
is:question is:answer |
| Exclude |
-[tag] -apples |
| For more details on advanced search visit our help page | |
Results tagged with policy-gradients
Search options not deleted
user 36821
For questions related to reinforcement learning algorithms often referred to as "policy gradients" (or "policy gradient algorithms"), which attempt to directly optimise a parameterised policy (without first attempting to estimate value functions) using gradients of an objective function with respect to the policy's parameters.
2
votes
What does it mean to parameterise a policy in policy gradient methods?
In the context of RL, for a policy to be parameterised it typically means we explicitly model the policy and is common in policy gradient methods.
Consider value based methods such as Q-learning whe …
- 5,030
3
votes
Why does (not) the distribution of states depend on the policy parameters that induce it?
The reason you are confused is because this is not the full derivation of the Policy Gradient Theorem. You are correct in thinking that $\mu(s)$ depends on the policy $\pi$ which in turn depends on th …
- 5,030
1
vote
Accepted
I have a few doubts understanding and implementing Proximal Policy Optimisation Algorithm
A rollout buffer is, generally speaking, a buffer for a much shorter time horizon than a replay buffer, and is discarded after use. The replay buffer in off-policy methods like DQN store all historica …
- 5,030
4
votes
Accepted
Why does REINFORCE work at all?
The key to REINFORCE working is the way the parameters are shifted towards $G \nabla \log \pi(a|s, \theta)$.
Note that $ \nabla \log \pi(a|s, \theta) = \frac{ \nabla \pi(a|s, \theta)}{\pi(a|s, \theta) …
- 5,030
2
votes
Accepted
How to calculate the advantage in policy gradient functions?
First let us note the definition of the advantage function:
$$A(s,a) = Q(s,a) - V(s) \; ,$$
where $Q(s,a)$ is the action-value function and $V(s)$ is the state-value function. In theory you could r …
- 5,030
2
votes
Why adding a baseline doesn't affect the policy gradient?
The policy gradient states that
$$\nabla J(\theta) \propto \sum_s \mu(s) \sum_a q_\pi(s, a) \nabla\pi(a | s; \theta)\;$$
where the derivatives are taken wrt the parameter $\theta$.
Now, if we say inco …
- 5,030
3
votes
Accepted
What is the loss for policy gradients with continuous actions?
This update rule can still be applied in the continuous domain.
As pointed out in the comments, suppose we are parameterising our policy using a Gaussian distribution, where our neural networks take a …
- 5,030
3
votes
Accepted
Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?
This is not quite the loss that is stated in the paper.
For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\t …
- 5,030
2
votes
Accepted
What specifically is the gradient of the log of the probability in policy gradient methods?
I would recommend not trying to think of this in relation to supervised learning.
The policy $\pi(\cdot; \theta)$ is simply a function that is parameterised by $\theta$. If we take a $\log$ of this fu …
- 5,030
3
votes
Accepted
Which loss function should I use in REINFORCE, and what are the labels?
The loss function you are looking for is cross entropy loss. The 'label' that you use is the action you took at the time point you are updating for.
- 5,030
1
vote
How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\thet...
When the authors write go from
$$\nabla_{\theta}J \propto \sum_s \mu(s) \sum_a q_{\pi}(s,a)\nabla_{\theta}\pi(a|s;\theta)\;$$
to
$$\nabla_{\theta}J = E_{\pi}\left[\sum_a q_{\pi}(S_t,a) \nabla_{\theta} …
- 5,030
1
vote
Is the policy gradient expression in Fundamentals of Deep Learning wrong?
There is no sign error and we should not change to $\arg\max$. With Policy Gradients I find that it is not useful to think about things such as a 'loss'.
In short, we want to first find the derivative …
- 5,030
1
vote
Accepted
Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the ...
You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.
Essentially when they take the expectation with respe …
- 5,030
3
votes
Accepted
How exactly is $Pr(s \rightarrow x, k, \pi)$ deduced by "unrolling", in the proof of the pol...
The unrolling step is due to the fact you end up with an equation that you can keep expanding indefinitely.
Note that we start with calculating $\nabla v_\pi(s)$ and arrive at
$$\nabla v_\pi(s) = \sum …
- 5,030
2
votes
Is the re-parameterization trick necessary in the policy gradient method?
For Policy-Gradient methods which require a differentiable sample from the action distribution, such as Soft Actor-Critic, then you are correct that it suffers from the same problem of requiring the g …
- 5,030