11
votes
How large should the replay buffer be?
You need to read this 2020 paper by Deepmind:
"Revisiting Fundamentals of Experience Replay"
They explicitly test the size of the experience replay, the replay-ratio of each experience and ...
- 280
10
votes
How large should the replay buffer be?
In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything.
The larger the ...
- 41.4k
5
votes
Why are reinforcement learning methods sample inefficient?
I will try to give a broad answer, if it's not helpful I'll remove it.
When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
- 5,428
4
votes
Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity
(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
- 171
4
votes
Why are reinforcement learning methods sample inefficient?
This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
- 514
3
votes
Accepted
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
You certainly can, that is indeed one of the advantages of off-policy learning ...
- 10.5k
3
votes
Accepted
Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?
This is not quite the loss that is stated in the paper.
For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\...
- 5,030
3
votes
Why is DDPG an off-policy RL algorithm?
DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
- 5,030
2
votes
Accepted
Why feed actions in later layer in Q network?
So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions?
Yes.
...
- 33.3k
2
votes
Is DDPG just for deterministic environments?
I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment.
Stochastic policy: ...
- 151
2
votes
Training actor-critic algorithms in games with opponents
The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying ...
- 10.5k
2
votes
Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?
I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method ...
- 81
2
votes
Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?
The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this ...
- 5,030
2
votes
Is there a good website where I can learn about Deep Deterministic Policy Gradient?
Spinning Up by Open Ai.
Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html
- 43
2
votes
Accepted
DDPG doesn't converge for MountainCarContinuous-v0 gym environment
I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge:
Sampled the noise from a standard normal distribution instead of ...
- 125
2
votes
How to have zero value or a value between 200 and 400 in the output of a deep learning model?
generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq ...
- 2,389
2
votes
Is it a bad practice to use cumulative rewards in reinforcement learning
Reinforcement learning already has the objective of maximising the sum of future expected reward.
By making each reward the sum of all previous rewards, you will make the the difference between good ...
- 33.3k
2
votes
Accepted
How to sample the tuples during the initial time steps of the DDPG algorithm?
You have a free choice to either:
Wait until the replay buffer hits a minimum size for sampling.
Take smaller samples from the buffer initially, until the buffer is large enough. On the first time ...
- 33.3k
1
vote
Replay buffer action range in DDPG
It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not ...
- 13
1
vote
Accepted
Why does my actor-critic network always give either -1 or 1 at the output layer?
Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
- 562
1
vote
Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)
Straight theoretical answer:
In theory, yes, it is possible to model this problem as a Reinforcement Learning. But in practice, RL is not the most suitable approach for a simple linear maximization ...
- 854
1
vote
Accepted
Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?
You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either.
Essentially when they take the expectation with ...
- 5,030
1
vote
Accepted
How does DDPG algorithm know about my action mapping function?
I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[...
- 5,030
1
vote
Accepted
What do the state features of KukaGymEnv represent?
Here's an incomplete answer, but it may help.
Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function <...
- 601
1
vote
Accepted
Using DDPG for control in multi-dimensional continuous action space?
First, is it even possible to use DDPG for multi-dimensional
continuous action spaces?
Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here....
- 36
1
vote
Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?
Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.
- 171
1
vote
In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?
Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.
- 1,136
1
vote
Accepted
What made your DDPG implementation on your environment work?
Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment:
Reducing the neural network size, compared to the original paper. Instead of:
2 hidden layers ...
- 722
1
vote
Accepted
How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?
After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG:
I added a reward component based on the actuator ...
- 171
Only top scored, non community-wiki answers of a minimum length are eligible