Questions tagged [experience-replay]
For questions related to the "experience replay" buffer (as used in the Deep Q Network and similar works).
50
questions
1
vote
0
answers
41
views
Prioritized experience replay correction with off-policy estimators
Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update.
Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size ...
- 153
1
vote
1
answer
79
views
DQN with experience history to learn from already saved - which reward should I take?
I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. ...
- 33
0
votes
0
answers
218
views
Confusion about N-step buffer in DQN
So I've been trying to implement the n-step buffer in DQN algorithm.
Temporal difference for which we score values $(s_i, a_i, s_{i + n}, r, d)$ is defined in this way
$$
\mathrm{TD}_{n}(s_i, a_i, s^{'...
0
votes
1
answer
77
views
Can TRPO use replay buffers?
I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.
Is it possible to ...
- 1
1
vote
0
answers
27
views
Why can't I train like a dataset of samples instead of maintaining replay buffer?
On observing the DDPG algorithm, we notice that the updation of neural networks is happening during the episode.
But, it seems there is no issue if we allow the completion of an episode and then treat ...
- 3,612
2
votes
1
answer
96
views
How to sample the tuples during the initial time steps of the DDPG algorithm?
I am facing an issue in understanding the following line from the pseudocode of the DDPG algorithm
Sample a random minibatch of $N$ transitions $(s_i, a_i, r_i, s_{i+1})$ from $R$
Here $N$ is a ...
- 3,612
1
vote
0
answers
34
views
Does deep RL techniques only interested in 'unit transitions' rather than 'whole experience'?
In deep-rl techniques, if I understand correctly, a replay buffer is used in training the neural networks. The purpose of using the replay buffer is to store the experience and send a (sampled) batch ...
- 3,612
3
votes
1
answer
486
views
Why is a large replay buffer inefficient?
Open AI spin up says
... the replay buffer should be large enough to contain a wide range
of experiences, but it may not always be good to keep everything. If
you only use the very-most recent data, ...
- 131
1
vote
1
answer
334
views
How to handled delayed rewards in contextual bandits [closed]
All the examples I see in the tf_Agents for contextual bandits, involves a reward function we generated the reward instantly after an observation has been generated.
But, in my real world usecase (say ...
- 111
1
vote
1
answer
201
views
Doesn't the n-step Tree Backup algorithm negatively affect the DQN-Agent by creating inconsistent look-ahead targets?
In the text book of Sutton and Barto on page 152 they introduce the n-step Tree Backup algorithm, where the tree-backup n-step return is defined via
$$
G_{t:t+n} = R_{t+1} + \gamma \sum_{a \neq A_{t+1}...
- 55
-1
votes
1
answer
175
views
What does the line of code "self.buffer[-1] = observation" do in this BufferWrapper class for DQN?
So the code is related to using a buffer
...
- 185
2
votes
2
answers
209
views
What is the purpose of storing the action $a$ within an experience tuple?
From what I understand, experience replay works by storing tuples of $(s, a, r, s')$ to be sampled for training. I understand why we store $s$, $r$ and $s'$. However, I do not understand the need for ...
1
vote
2
answers
703
views
Can stochastic gradient descent be properly used in any sample based learning algorithm in Reinforcement Learning?
Assuming we use an MSE cost function of the form
$$ \sum_s\mu(s)(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2 = E_{\mu(s)}[(V_{\pi}(S_t)-\hat{V}(S_t,\theta_t))^2])$$
The Stochastic Gradient Descent is used ...
- 384
2
votes
1
answer
336
views
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
4
votes
0
answers
99
views
Where does this variation of the importance sampling weight come from?
I have seeing a variation in importance sampling (IS) in Prioritized Experience Replay (PER) in some implementations regarding the original paper approach stated as (in section 3.4):
$$
w_{i}=\left(\...
- 81
0
votes
0
answers
32
views
What are the implications of storing the alternative situation (that could have been experienced) in the replay buffer?
Consider an environment where there are 2 outcomes (e.g. dead and alive) and a discrete set of actions. For example, a game where the agent has 2 guns $A$ and $B$ to shoot a monster (the monster dies ...
- 11
3
votes
0
answers
59
views
Why is it necessary to divide the priority range according to the batch size in Prioritized Experience Replay?
According to DeepMinds's paper Prioritized Experience Replay (2016), specifically Appendix B.2.1 "Proportional prioritization" (p. 13), one should equally divide the priority range $[0, p_\...
- 31
3
votes
1
answer
364
views
Why is sampling non-uniformly from the replay memory an issue? (Prioritized experience replay)
I can't seem to understand why we need importance sampling in prioritized experience replay (PER). The authors of the paper write on page 5:
The estimation of the expected value with stochastic ...
- 43
5
votes
1
answer
730
views
Why do DQNs tend to forget?
Why do DQNs tend to forget? Is it because when you feed highly correlated samples, your model (function approximation) doesn't give a general solution?
For example:
I use level 1 experiences, my ...
- 359
2
votes
1
answer
135
views
What is the advantage of using experience replay (as opposed to feeding it sequential data)?
Let's suppose that our RL agent needs to play a game with different levels. If we train our RL agent sequentially or with sequential data, our agent will learn how to play level 1, but then it will ...
- 359
2
votes
0
answers
59
views
Prioritised Remembering in Experience Replay (Q-Learning)
I'm using Experience Replay based on the original Prioritized Experience Replay (PER) paper. In the paper authors show ~ an order of magnitude increase in data efficiency from prioritized sampling. ...
1
vote
1
answer
299
views
Do we need multiple parallel environments to train in batches an on-policy algorithm?
When using an on-policy method in reinforcement learning, like advantage actor-critic, you shouldn't use old data from an experience buffer, since a new policy requires new data. Does this mean that ...
- 111
3
votes
1
answer
369
views
How to handle the final state in experience replay?
I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last ...
- 763
1
vote
1
answer
244
views
What would happen if we sampled only one tuple from the experience replay?
The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right?
What would happen if we calculate our ...
- 359
0
votes
0
answers
162
views
My Double DQN with Experience Replay produces a no-action decision most of the time. Why?
I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...
- 31
2
votes
0
answers
67
views
In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?
I've written a Double DQN-based stock trading bot using mainly time series stock data.
I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
- 31
5
votes
1
answer
982
views
How does being on-policy prevent us from using the replay buffer with the policy gradients?
One of the approaches to improving the stability of the Policy
Gradient family of methods is to use multiple environments in
parallel. The reason behind this is the fundamental problem we
discussed in ...
- 261
2
votes
1
answer
143
views
Could we update the policy network with previous trajectories using supervised learning?
I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
- 451
3
votes
1
answer
235
views
Can experience replay be used for training after completing every single epoch?
The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
- 33
2
votes
1
answer
77
views
Intutitive explanation of why Experience Replay is used in a Deep Q Network?
I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the ...
- 309
1
vote
0
answers
239
views
Implementing Actor-Critic with Experience Replay for Continuous Action Spaces
I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here:
Sample Efficient Actor-Critic with Experience ...
- 11
1
vote
0
answers
29
views
Can I apply experience on naive actor critic directly? Should it work?
Can I apply experience replay on naive actor-critic directly? Should it work?
I have tried that but unfortunately it didn't work.
- 11
3
votes
0
answers
179
views
What is the difference between random and sequential sampling from the reply memory?
I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.
...
- 133
4
votes
1
answer
124
views
What does the notation $p_t = \text{max}_{i<t} p_i$ mean in algorithm 1 of the prioritized experience replay paper?
I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below):
I understand that new transitions (not visited before) ...
- 519
3
votes
1
answer
116
views
Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?
In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below):
Why do the authors track this at every time step? ...
- 519
7
votes
2
answers
1k
views
Which kind of prioritized experience replay should I use?
The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
- 2,028
1
vote
0
answers
629
views
Do we need to use the experience replay buffer with the A3C algorithm?
I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm.
The approached I used is the following:
...
- 61
4
votes
1
answer
305
views
Experience Replay Not Always Giving Better Results
I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
2
votes
0
answers
373
views
Why experience reply memory in DQN instead of a RNN memory?
I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
- 1,036
14
votes
2
answers
14k
views
How large should the replay buffer be?
I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written
In order for the algorithm to have stable behavior, the replay buffer should ...
- 341
1
vote
0
answers
286
views
Reinforcement Learning with limited number of episodes
I try to implement RL to a case something like this:
This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different ...
- 2,789
1
vote
0
answers
43
views
Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?
According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
- 451
14
votes
3
answers
6k
views
Why exactly do neural networks require i.i.d. data?
In reinforcement learning, successive states (actions and rewards) can be correlated. An experience replay buffer was used, in the DQN architecture, to avoid training the neural network (NN), which ...
- 39.5k
1
vote
2
answers
154
views
When are Q values calculated in experience replay?
In experience replay, the update rule follows the loss:
$$
L_i(\theta_i) = \mathbb{E}_{(s_t, a_t, r_t, s_{t+1}) \sim U(D)} \left[ \left(r_t + \gamma \max_{a_{t+1}} Q(s_{t+1}, a_{t+1}; \theta_i^-) - Q(...
- 749
3
votes
1
answer
491
views
What information should be cached in experience replay for actor-critic?
Experience replay is buffer (or a "memory") of transactions $e_t = (s_t, a_t, r_t, s_{t+1})$.
The equations for calculating the loss in actor critic are an
actor loss (parameterized by $\theta$) $$...
- 749
8
votes
1
answer
314
views
Is Experience Replay like dreaming?
Drawing parallels between Machine Learning techniques and a human brain is a dangerous operation. When it is done successfully, it can be a powerful tool for vulgarisation, but when it is done with no ...
- 591
7
votes
1
answer
161
views
In imitation learning, do you simply inject optimal tuples of experience $(s, a, r, s')$ into your experience replay buffer?
Due to my RL algorithm having difficulties learning some control actions, I've decided to use imitation learning/apprenticeship learning to guide my RL to perform the optimal actions. I've read a few ...
- 433
18
votes
1
answer
18k
views
How does LSTM in deep reinforcement learning differ from experience replay?
In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are:
How does this method differ from the ...
- 353
3
votes
1
answer
425
views
When using experience replay, do we update the parameters for all samples of the mini-batch or for each sample in the mini-batch separately?
I've been reading Google's DeepMind Atari paper and I'm trying to understand how to implement experience replay.
Do we update the parameters $\theta$ of function $Q$ once for all the samples of the ...
- 241
8
votes
2
answers
917
views
What is experience replay in laymen's terms?
I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...
- 241