# Tag Info

### How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
• 33.7k

### How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" Also, to add to the answer by @nbro Assume you implement experience replay as a buffer where the ...
• 200

### Why are reinforcement learning methods sample inefficient?

I will try to give a broad answer, if it's not helpful I'll remove it. When we talk about sampling we are actually talking about the number of interaction required to an agent to learn a good model ...
• 3,503

### Why are reinforcement learning methods sample inefficient?

This is mostly because humans already have information when they start learning the game (priors) that makes them learn it more quickly. We already know to jump on monsters or avoid them or to get ...
Accepted

### Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
• 9,357

### Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
• 4,055
Accepted

### Why feed actions in later layer in Q network?

So does that mean, that the input of the first hidden layer was simply the state and the input of the second hidden layer the output of the first hidden layer concatenated with the actions? Yes. ...
• 23.8k

### Is DDPG just for deterministic environments?

I am not an expert in this area. But I believe that the word "Deterministic" is for "Policy" in the "Deterministic Policy" Gradient. It does not mean deterministic environment. Stochastic policy: ...
• 141

### Appropriate algorithm for RL problem with sparse rewards, continuous actions and significant stochasticity

(1) You might want look into RND (Random network distillation) which allows usage of a curiosity-based exploration bonus for the agent as an intrinsic reward. You can use the intrinsic reward to ...
• 151

### Is there a good website where I can learn about Deep Deterministic Policy Gradient?

Spinning Up by Open Ai. Be sure to read up Part 3 (Intro to Policy Optimisation) before you move on to : https://spinningup.openai.com/en/latest/algorithms/ddpg.html
• 43
Accepted

• 2,249
1 vote
Accepted

### Why is the behaviour policy denoted by $\beta$ and the exploration policy by $\mu'$ in the DDPG paper?

You are right, it is sloppy notation by the authors. However, the target network is not necessarily linked to the behaviour policy $\beta$ either. Essentially when they take the expectation with ...
• 4,055
1 vote

### Replay buffer action range in DDPG

It has an obvious answer: Network is conditioned to use tanh activation. Hence the buffer should be [-1, 1], or unscaled values before action execution. As I am not using openai gym or other baselines ...
1 vote
Accepted

### Why does my actor-critic network always give either -1 or 1 at the output layer?

Most probably your network is underfitted. In that case, the network outputs values randomly. Hyperbolic tangent tanh converges very quickly towards $-1$ or $1$, so that is why you always find $-1$ ...
• 512
1 vote

### Is it a bad practice to use cumulative rewards in reinforcement learning

Reinforcement learning already has the objective of maximising the sum of future expected reward. By making each reward the sum of all previous rewards, you will make the the difference between good ...
• 23.8k
1 vote

### Is it possible to solve a linear programming problem using reinforcement learning? (DDPG algorithm)

Straight theoretical answer: In theory, yes, it is possible to model this problem as a Reinforcement Learning. But in practice, RL is not the most suitable approach for a simple linear maximization ...
1 vote

• 4,055
1 vote
Accepted

### What do the state features of KukaGymEnv represent?

Here's an incomplete answer, but it may help. Your state is read by the function getExtendedObservation(). This function makes two things : it calls the function <...
• 571
1 vote
Accepted

### Using DDPG for control in multi-dimensional continuous action space?

First, is it even possible to use DDPG for multi-dimensional continuous action spaces? Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here....
• 36
1 vote
Accepted

### DDPG doesn't converge for MountainCarContinuous-v0 gym environment

I had to change the actions selection function for this and tune some hyper-parameters. Here's what I did to make it converge: Sampled the noise from a standard normal distribution instead of ...
1 vote
Accepted

Below are some tweaks that helped me accelerate the training of DDPG on a Reacher-like environment: Reducing the neural network size, compared to the original paper. Instead of: 2 hidden layers ...
• 682
1 vote

### Purpose of using actor-critic algorithms under deterministic MDP dynamics?

In deep reinforcement learning for portfolio optimization, many researchers (Xiong at al. for example) use historical market data for model training. The resulting MDP dynamics is of course completely ...
• 23.8k
1 vote

### Training actor-critic algorithms in games with opponents

The specific approaches you mentioned (A3C, DDPG), and usually also other Actor-Critic methods in general, are approaches for the standard single-agent Reinforcement Learning (RL) setting. When trying ...
• 9,357
1 vote

### Why Q2 is a more or less independant estimate in Twin Delayed DDPG (TD3)?

I emailed the author of the paper and he replied that randomness in the parameter initialization is the only difference between Q1 and Q2. This difference is enough in practice. Moreover, TD3 method ...

Only top scored, non community-wiki answers of a minimum length are eligible