17 votes
Accepted

How does LSTM in deep reinforcement learning differ from experience replay?

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one ...
user avatar
  • 23.8k
11 votes

Why exactly do neural networks require i.i.d. data?

There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - ...
user avatar
  • 23.8k
9 votes

How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
user avatar
  • 33.7k
8 votes
Accepted

Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
user avatar
  • 23.8k
8 votes

Why exactly do neural networks require i.i.d. data?

Suppose that we have some optimization criterion $J(x)$, which we aim to optimize (maybe maximize, maybe minimize), which we can compute for a single example $x$. In an "ideal world", where we have ...
user avatar
  • 9,357
7 votes

How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" Also, to add to the answer by @nbro Assume you implement experience replay as a buffer where the ...
user avatar
  • 200
6 votes

Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
user avatar
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
user avatar
  • 9,357
4 votes
Accepted

How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|...
user avatar
  • 2,226
4 votes
Accepted

What information should be cached in experience replay for actor-critic?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. ...
user avatar
4 votes

What is experience replay in laymen's terms?

In reinforcement learning (RL), an agent interacts with an environment in time steps. At each time step $t$, the agent and the environment are in some state $s_t$. From that state $s_t$, the agent ...
user avatar
  • 33.7k
4 votes
Accepted

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
user avatar
  • 9,357
4 votes
Accepted

Why do DQNs tend to forget?

You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.
user avatar
  • 303
3 votes
Accepted

What is the purpose of storing the action $a$ within an experience tuple?

We need to store the action $a$ as it tells us the action that we took in the state that we are backing up. Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and ...
user avatar
3 votes

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many ...
user avatar
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
user avatar
  • 9,357
3 votes
Accepted

What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i $ means that the priority of transition number $t$ will be the ...
user avatar
2 votes

In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?

That seems to be functional. That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-...
user avatar
2 votes
Accepted

When are Q values calculated in experience replay?

Because the Q value is different, I don't see how the reward signal at time $t$ is of any relevance for $Q_{t+x}(s_t,a_t)$ at $t+x$, the time of learning. The $r_t$ value for any single step is not ...
user avatar
  • 23.8k
2 votes
Accepted

Could we update the policy network with previous trajectories using supervised learning?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
user avatar
  • 2,226
2 votes
Accepted

Can experience replay be used for training after completing every single epoch?

My question is, if I am to play the game 10000 epochs, store all the experiences and then train from the experiences would that have the same effect as training and while running through 10000 epochs? ...
user avatar
  • 23.8k
2 votes
Accepted

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

It is the neural network approximation that suffers, when it attempts to learn from correlated data. Intuitively, this is because the learning algorithm takes gradient steps assuming that the examples ...
user avatar
  • 23.8k
2 votes
Accepted

How to handle the final state in experience replay?

You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn. However, ...
user avatar
  • 23.8k
2 votes
Accepted

Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)

The problem is not that we need importance sampling because the learning is off-policy -- you are correct in that for one step off-policy algorithms such as $Q$-learning we don't need importance ...
user avatar
2 votes

What is the purpose of storing the action $a$ within an experience tuple?

The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to ...
user avatar
2 votes

How to handled delayed rewards in contextual bandits

The update rules are not any different. However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp. ...
user avatar
  • 23.8k
2 votes

Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
user avatar
1 vote
Accepted

What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and experience replay), but that code should be very clear to ...
user avatar
  • 33.7k
1 vote

Do we need multiple parallel environments to train in batches an on-policy algorithm?

We don't need multiple environments. On-policy algorithms require that new training samples are collected with the newest policy, so we can't use an experience buffer. However we can use the newest ...
user avatar
  • 163
1 vote

Which kind of prioritized experience replay should I use?

It appears that the rank based method would be slightly better in terms of time complexity because you sort only once for k number of sampling operations. This article (not mine) explains in detail ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible