17 votes
Accepted

How does LSTM in deep reinforcement learning differ from experience replay?

How does this method differ from the experience replay, as they both use past information in the training? What's the typical application of both techniques? Using a recurrent neural network is one ...
Neil Slater's user avatar
  • 31.7k
11 votes

Why exactly do neural networks require i.i.d. data?

There is an assumption behind the theory training a neural network, that also applies to many other supervised learning methods, that a training sample is representative of the data set as a whole - ...
Neil Slater's user avatar
  • 31.7k
10 votes

How large should the replay buffer be?

You need to read this 2020 paper by Deepmind: "Revisiting Fundamentals of Experience Replay" They explicitly test the size of the experience replay, the replay-ratio of each experience and ...
Kari's user avatar
  • 270
10 votes

How large should the replay buffer be?

In order for the algorithm to have stable behavior, the replay buffer should be large enough to contain a wide range of experiences, but it may not always be good to keep everything. The larger the ...
nbro's user avatar
  • 40.2k
8 votes

Why exactly do neural networks require i.i.d. data?

Suppose that we have some optimization criterion $J(x)$, which we aim to optimize (maybe maximize, maybe minimize), which we can compute for a single example $x$. In an "ideal world", where we have ...
Dennis Soemers's user avatar
  • 10.2k
8 votes
Accepted

Is Experience Replay like dreaming?

The speaker argued that a dream is a random addition of memories, just as experience replay. The speaker is taking some liberties due to a general lack of scientific understanding of what dreams are. ...
Neil Slater's user avatar
  • 31.7k
6 votes

Which kind of prioritized experience replay should I use?

The authors of that paper hypothesized that rank-based prioritization would be more robust to outliers. They suggested that rank-based sampling would be preferred for this reason. However, as they ...
Philip Raeisghasem's user avatar
5 votes
Accepted

Why do DQNs tend to forget?

You are referring to catastrophic forgetting which could be an issue in any neural net. More specifically for DQN refer to this article.
pedrum's user avatar
  • 313
5 votes
Accepted

When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?

Gradient descent should be performed using the sum (or average) of the losses in the minibatch. This is in fact also how I read the pseudocode in your question, though I understand it can be confusing....
Dennis Soemers's user avatar
  • 10.2k
5 votes
Accepted

How does being on-policy prevent us from using the replay buffer with the policy gradients?

Let's say your old policy is $\pi_b$ and your current one is $\pi_a$. If you collected trajectory by using policy $\pi_b$ you would get return $G$ whose expected value is \begin{align} E_{\pi_b}[G_t|...
Brale's user avatar
  • 2,366
4 votes
Accepted

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In some cases we may wish to have a discount factor $\gamma_t$ which depends on time $t$ (or depends on state $s_t$ and/or action $a_t$, leading to an indirect dependence on time $t$). Indeed we do ...
Dennis Soemers's user avatar
  • 10.2k
4 votes
Accepted

What information should be cached in experience replay for actor-critic?

The loss function is estimated in every batch training cycle. Gradients of the loss are computed and propagation back to the network happens in every cycle. This means that you use a small batch (e.g. ...
Constantinos's user avatar
4 votes

What is experience replay in laymen's terms?

In reinforcement learning (RL), an agent interacts with an environment in time steps. At each time step $t$, the agent and the environment are in some state $s_t$. From that state $s_t$, the agent ...
nbro's user avatar
  • 40.2k
3 votes
Accepted

What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?

From my interpretation what it means is that $p_t$ is the priority value associated with each transition and $p_t = max_{i<t} p_i $ means that the priority of transition number $t$ will be the ...
Miguel Saraiva's user avatar
3 votes

Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?

First I will address the issue of Tabular methods. These do not use SGD at all. Although the updates are very similar to an SGD update there is no gradient here and so we are not using SGD. Many ...
David's user avatar
  • 4,770
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
Dennis Soemers's user avatar
  • 10.2k
3 votes
Accepted

What is the purpose of storing the action $a$ within an experience tuple?

We need to store the action $a$ as it tells us the action that we took in the state that we are backing up. Suppose we are in state $s$ and we take action $a$, then we will receive a reward $r$ and ...
David's user avatar
  • 4,770
3 votes

In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?

That seems to be functional. That is a great approach, as long as you are using an off-policy algorithm (since the samples you are using to learn are not the policy currently being performed), like Q-...
BlueMoon93's user avatar
2 votes
Accepted

When are Q values calculated in experience replay?

Because the Q value is different, I don't see how the reward signal at time $t$ is of any relevance for $Q_{t+x}(s_t,a_t)$ at $t+x$, the time of learning. The $r_t$ value for any single step is not ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

Could we update the policy network with previous trajectories using supervised learning?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
Brale's user avatar
  • 2,366
2 votes
Accepted

Can experience replay be used for training after completing every single epoch?

My question is, if I am to play the game 10000 epochs, store all the experiences and then train from the experiences would that have the same effect as training and while running through 10000 epochs? ...
Neil Slater's user avatar
  • 31.7k
2 votes
Accepted

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

It is the neural network approximation that suffers, when it attempts to learn from correlated data. Intuitively, this is because the learning algorithm takes gradient steps assuming that the examples ...
Neil Slater's user avatar
  • 31.7k
2 votes

What is the purpose of storing the action $a$ within an experience tuple?

The goals of experience replay as first proposed by Lin (1992) and more recently applied successfully in the DQN algorithm by Mnih et al. (2013) are to break temporal correlations of updates and to ...
Floris den Hengst's user avatar
2 votes
Accepted

Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)

The problem is not that we need importance sampling because the learning is off-policy -- you are correct in that for one step off-policy algorithms such as $Q$-learning we don't need importance ...
David's user avatar
  • 4,770
2 votes
Accepted

How to handle the final state in experience replay?

You do not store a terminal state as $s$ in the replay table because by definition its value is always $0$, and there is no action, reward or next state. There is literally nothing to learn. However, ...
Neil Slater's user avatar
  • 31.7k
2 votes

How to handled delayed rewards in contextual bandits

The update rules are not any different. However, if you make many other decisions in the meantime, the timestamps that you are able to run estimate updates for will lag behind the current timestamp. ...
Neil Slater's user avatar
  • 31.7k
2 votes

Why is a large replay buffer inefficient?

I read the same thing recently, and my interpretation was this: If you only use the very-most recent data, you will overfit to that and things will break We'd like to train the network to predict ...
Lee Reeves's user avatar
2 votes
Accepted

How to sample the tuples during the initial time steps of the DDPG algorithm?

You have a free choice to either: Wait until the replay buffer hits a minimum size for sampling. Take smaller samples from the buffer initially, until the buffer is large enough. On the first time ...
Neil Slater's user avatar
  • 31.7k
1 vote
Accepted

What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?

I don't know if you're confused about this code because you're not very familiar with Python or reinforcement learning (specifically, DQN and experience replay), but that code should be very clear to ...
nbro's user avatar
  • 40.2k
1 vote

What is the advantage of using experience replay (as opposed to feeding it sequential data)?

Random sampling from the replay memory helps us reach closer to the i.i.d assumption and also lets the learning algorithm train on the data multiple times compared to just once. Therefore, there is a ...
desert_ranger's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible