Questions tagged [experience-replay]

For questions related to the "experience replay" buffer (as used in the Deep Q Network and similar works).

Filter by
Sorted by
Tagged with
3
votes
1answer
36 views

How to handle the final state in experience replay?

I'm using the DQN algorithm to train my agent to play a turn-based game. The memory replay buffer stores tuples of experiences $(s, a, r, s')$, where $s$ and $s'$ are consecutive states. At the last ...
1
vote
1answer
25 views

What would happen if we sampled only one tuple from the experience replay?

The concept of experience replay is saving our experiences in our replay buffer. We select at random to break the correlation between consecutive samples, right? What would happen if we calculate our ...
0
votes
0answers
16 views

My Double DQN with Experience Replay produces a no-action decision most of the time. Why?

I've written a Double DQN-based stock trading bot using mainly time series stock data. The internal network of the Double DQN is a LSTM which handles the time series data. An Experience Replay buffer ...
1
vote
0answers
18 views

In a DQN, can Prioritized Experience Replay actually perform worse than a regular Experience Replay?

I've written a Double DQN-based stock trading bot using mainly time series stock data. I've recently upgraded my Experience Replay(ER) code with a version of Prioritized Experience Replay (PER) ...
2
votes
1answer
67 views

On-policy preventing us from using the replay buffer with the PG?

One of the approaches to improving the stability of the Policy Gradient family of methods is to use multiple environments in parallel. The reason behind this is the fundamental problem we ...
2
votes
1answer
36 views

Could we update the policy network with previous trajectories using supervised learning?

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
3
votes
0answers
35 views

Is this a good approach to solving Atari's “Montezuma's Revenge”?

I'm new to Reinforcement Learning. For an internship, I am currently training Atari's "Montezuma's Revenge" using a double Deep Q-Network with Hindsight Experience Replay (HER). HER is supposed to ...
2
votes
1answer
35 views

How does the optimization process in hindsight experience replay exactly work?

I was reading the following research paper Hindsight Experience Replay. This is the paper that introduces a concept called Hindsight Experience Replay (HER), which basically attempts to alleviate the ...
3
votes
1answer
42 views

Can experience replay be used for training after completing every single epoch?

The DQN implements replay memory. Based on my research, I believe the replay memory starts to get used for training once there is enough experience in the memory buffer. This means the neural network ...
2
votes
1answer
36 views

Intutitive explanation of why Experience Replay is used in a Deep Q Network?

I understand that Experience Replay is used for data efficiency reasons and to remove correlations in sequences of data. How exactly do these sequences of correlated data affect the performance of the ...
1
vote
0answers
63 views

Implementing Actor-Critic with Experience Replay for Continuous Action Spaces

I have been trying to implement the ACER algorithm for continuous action spaces in reinforcement learning. The paper for the algorithm can be found here: Sample Efficient Actor-Critic with Experience ...
1
vote
0answers
17 views

Can I apply experience on naive actor critic directly? Should it work?

Can I apply experience replay on naive actor-critic directly? Should it work? I have tried that but unfortunately it didn't work.
3
votes
0answers
39 views

What is the difference between random and sequential sampling from the reply memory?

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure. ...
2
votes
1answer
61 views

New transition priorities in Prioritized Experience Replay?

I am having a hard time converting line 6 of the prioritized experience replay algorithm from the original paper into plain English (see below): I understand that new transitions (not visited before) ...
3
votes
1answer
74 views

Why do authors track $\gamma_t$ in Prioritized Experience Replay Paper?

In the original prioritized experience replay paper, the authors track $\gamma_t$ in every state transition tuple (see line 6 in algorithm below): Why do the authors track this at every time step? ...
4
votes
1answer
288 views

Which kind of prioritized experience replay should I use?

The Prioritized Experience Replay paper gives two different ways of sampling from the replay buffer. One, called "proportional prioritization", assigns each transition a priority proportional to its ...
1
vote
0answers
187 views

Do we need to use the experience replay buffer with the A3C algorithm?

I have skimmed through a bunch of deep learning books, but I have not yet understood whether we must use the experience replay buffer with the A3C algorithm. The approached I used is the following: ...
3
votes
1answer
87 views

Experience Replay Not Always Giving Better Results

I have recently started working on a control problem using a Deep Q Network as proposed by DeepMind (https://arxiv.org/abs/1312.5602). Initially, I implemented it without Experience Replay. The ...
2
votes
0answers
64 views

Why experience reply memory in DQN instead of a RNN memory?

I was trying to implement a DQN without experience reply memory, and the agent is not learning anything at all. I know from readings that experience reply is used for stabilizing gradients. But how ...
3
votes
2answers
2k views

How large should the replay buffer be?

I'm learning DDPG algorithm by following the following link: Open AI Spinning Up document on DDPG, where it is written In order for the algorithm to have stable behavior, the replay buffer should ...
1
vote
0answers
74 views

Reinforcement Learning with limited number of episodes

I try to implement RL to a case something like this: This game consist of several rounds. Every round the players need to generate a maze that consists of rooms. There are around 1000 different ...
1
vote
0answers
29 views

Should we multiply the target of actor by the importance sampling ratio when prioritized replay is applied to DDPG?

According to PER, we have to multiply the $Q$ error $\delta_i$ by the importance sampling ratio to correct the bias introduced by the imbalance sampling of PER, where importance sampling ratio is ...
2
votes
1answer
242 views

What information should be cached in experience replay for actor-critic?

Experience replay is buffer (or a "memory") of transactions $e_t = (s_t, a_t, r_t, s_{t+1})$. The equations for calculating the loss in actor critic are an actor loss (parameterized by $\theta$) $$...
14
votes
1answer
9k views

How does LSTM in deep reinforcement learning differ from experience replay?

In the paper Deep Recurrent Q-Learning for Partially Observable MDPs, the author processed the Atari game frames with an LSTM layer at the end. My questions are: How does this method differ from the ...
6
votes
2answers
393 views

What is experience replay in laymen's terms?

I've been reading Google's DeepMind Atari paper and I'm trying to understand the concept of "experience replay". Experience replay comes up in a lot of other reinforcement learning papers (...