I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.
Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.
Some places like this extract random (non-sequential) batches from the memory to train the neural network but Andrej Karpathy uses the sequential data to train the network.
Can someone tell me why there's the difference?