Hot answers tagged

9 votes

How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
user avatar
  • 33.7k
7 votes

How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" Also, to add to the answer by @nbro Assume you implement experience replay as a buffer where the ...
user avatar
  • 200
5 votes

Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
user avatar
4 votes

Why are reinforcement learning methods sample inefficient?

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
user avatar
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
user avatar
  • 9,357
3 votes

Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
user avatar
2 votes
Accepted

Why feed actions in later layer in Q network?

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions? Yes. ...
user avatar
  • 23.8k
2 votes

Is DDPG just for deterministic environments?

I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment. Stochastic policy: ...
user avatar
2 votes

Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
user avatar
  • 151
2 votes

Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Spinning Up by Open Ai. Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html
user avatar
  • 43
2 votes
Accepted

Why is the policy loss the mean of $-Q(s, \mu(s))$ in the DDPG algorithm?

This is not quite the loss that is stated in the paper. For standard policy gradient methods the objective is to maximise $v_{\pi_\theta}(s_0)$ -- note that this is analogous to minimising $-v_{\pi_\...
user avatar
2 votes

How to have zero value or a value between 200 and 400 in the output of a deep learning model?

generally the approach is to have a separate head. For example, imagine you have latent vector $z_k$, you would output two values: $h(z_k)$ and $f(z_k)$ where $0 \leq h \leq 1$ and $b_0 \leq f \leq ...
user avatar
  • 2,249
1 vote
Accepted

Why is the behaviour policy denoted by $\beta$ and the exploration policy by $ \mu'$ in the DDPG paper?

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either. Essentially when they take the expectation with ...
user avatar
1 vote

Replay buffer action range in DDPG

It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
user avatar
1 vote
Accepted

Why does my actor-critic network always give either -1 or 1 at the output layer?

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
user avatar
  • 512
1 vote

Is it a bad practice to use cumulative rewards in reinforcement learning

Reinforcement learning already has the objective of maximising the sum of future expected reward. By making each reward the sum of all previous rewards, you will make the the difference between good ...
user avatar
  • 23.8k
1 vote

Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

Straight theoretical answer: In theory, yes, it is possible to model this problem as a Reinforcement Learning. But in practice, RL is not the most suitable approach for a simple linear maximization ...
user avatar
1 vote

How does DDPG algorithm know about my action mapping function?

I would recommend doing is allowing your network to output any real number and then clipping the output. For instance, I was working with an agent that had to learn an angle between $[0, 2\pi]$ and $[...
user avatar
1 vote

Are there examples of agents that use a more modest number of parameters on Pendulum (or similar environments)?

Actually, I just started inspecting the entries further down in the leaderboard list, and there are in fact more modest architectures, e.g. this one, which uses 3 hidden layers with 8 units each.
user avatar
  • 151
1 vote

Understand the DDPG algorithm in Keras

The answer to your first question is because the line 'update the critic by minimising the loss $L = \frac{1}{N} \sum_i \left( y_i - Q(s_i, a_i |\theta^Q)\right)^2$ is implying that you will do this ...
user avatar
1 vote
Accepted

What do the state features of KukaGymEnv represent?

Here's an incomplete answer, but it may help. Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function <...
user avatar
  • 571
1 vote
Accepted

Using DDPG for control in multi-dimensional continuous action space?

First, is it even possible to use DDPG for multi-dimensional continuous action spaces? Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here....
user avatar
  • 36
1 vote
Accepted

DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge: Sampled the noise from a standard normal distribution instead of ...
user avatar
1 vote
Accepted

What made your DDPG implementation on your environment work?

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment: Reducing the neural network size, compared to the original paper. Instead of: 2 hidden layers ...
user avatar
1 vote

Purpose of using actor-critic algorithms under deterministic MDP dynamics?

In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely ...
user avatar
  • 23.8k
1 vote

Training actor-critic algorithms in games with opponents

The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying ...
user avatar
  • 9,357
1 vote

Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible