Hot answers tagged

10 votes

How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" They explicitly test the size of the experience replay, the replay-ratio of each experience and ...
Kari's user avatar
  • 270
10 votes

How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
nbro's user avatar
  • 40.2k
5 votes

Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
Edoardo Guerriero's user avatar
4 votes

Why are reinforcement learning methods sample inefficient?

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
Lustwelpintje's user avatar
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
Dennis Soemers's user avatar
  • 10.2k
3 votes
Accepted

Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?

This is not quite the loss that is stated in the paper. For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\...
David's user avatar
  • 4,770
3 votes

Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
David's user avatar
  • 4,770
3 votes

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
Huan's user avatar
  • 161
2 votes
Accepted

Why feed actions in later layer in Q network?

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions? Yes. ...
Neil Slater's user avatar
  • 31.7k
2 votes

Is DDPG just for deterministic environments?

I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment. Stochastic policy: ...
S. Phil Kim's user avatar
2 votes

Training actor-critic algorithms in games with opponents

The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying ...
Dennis Soemers's user avatar
  • 10.2k
2 votes

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method ...
Luke Guye's user avatar
2 votes

Why does this Keras implementation of the DDPG algorithm update the critic's network using the gradient but the pseudocode doesn't?

The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this ...
David's user avatar
  • 4,770
2 votes

Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Spinning Up by Open Ai. Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html
kikram's user avatar
  • 43
2 votes
Accepted

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge: Sampled the noise from a standard normal distribution instead of ...
Vedant Shah's user avatar
2 votes

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq ...
mshlis's user avatar
  • 2,359
2 votes

Is it a bad practice to use cumulative rewards in reinforcement learning

Reinforcement learning already has the objective of maximising the sum of future expected reward. By making each reward the sum of all previous rewards, you will make the the difference between good ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

How to sample the tuples during the initial time steps of the DDPG algorithm?

You have a free choice to either: Wait until the replay buffer hits a minimum size for sampling. Take smaller samples from the buffer initially, until the buffer is large enough. On the first time ...
Neil Slater's user avatar
  • 31.7k
1 vote

Replay buffer action range in DDPG

It has an obvious answer: Network is conditioned to use tanh activation. Hence the action values in the buffer should be in the range [-1, 1], or unscaled values before action execution. As I am not ...
goldfinch's user avatar
1 vote
Accepted

Why does my actor-critic network always give either -1 or 1 at the output layer?

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
devidduma's user avatar
  • 552
1 vote

Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

Straight theoretical answer: In theory, yes, it is possible to model this problem as a Reinforcement Learning. But in practice, RL is not the most suitable approach for a simple linear maximization ...
Andre Goulart's user avatar
1 vote
Accepted

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either. Essentially when they take the expectation with ...
David's user avatar
  • 4,770
1 vote
Accepted

How does DDPG algorithm know about my action mapping function?

I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[...
David's user avatar
  • 4,770
1 vote
Accepted

What do the state features of KukaGymEnv represent?

Here's an incomplete answer, but it may help. Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function <...
16Aghnar's user avatar
  • 601
1 vote
Accepted

Using DDPG for control in multi-dimensional continuous action space?

First, is it even possible to use DDPG for multi-dimensional continuous action spaces? Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here....
BAKYAC's user avatar
  • 36
1 vote

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.
Kris's user avatar
  • 171
1 vote

In Deep Deterministic Policy Gradient, are all weights of the policy network updated with the same or different value?

Each Q output is a scalar, so the sum of all those is a scalar. Thus, you're taking a gradient wrt your parameters of a scalar. The result is a vector with one entry per parameter.
harwiltz's user avatar
  • 1,126
1 vote
Accepted

What made your DDPG implementation on your environment work?

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment: Reducing the neural network size, compared to the original paper. Instead of: 2 hidden layers ...
user5093249's user avatar
1 vote
Accepted

How to avoid rapid actuator movements in favor of smooth movements in a continuous space and action space problem?

After some research on the subject, I found a possible solution to my problem of high frequency oscillations in continuous control using DDPG: I added a reward component based on the actuator ...
opt12's user avatar
  • 171

Only top scored, non community-wiki answers of a minimum length are eligible