3
$\begingroup$

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.

Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.

Some places like this extract random (non-sequential) batches from the memory to train the neural network but Andrej Karpathy uses the sequential data to train the network.

Can someone tell me why there's the difference?

Improve this question
$\endgroup$
4
  • $\begingroup$ You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm $\endgroup$ – Brale Sep 20 '19 at 10:42
  • $\begingroup$ What do you mean? $\endgroup$ – Sarvagya Gupta Sep 20 '19 at 12:17
  • 1
    $\begingroup$ First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it $\endgroup$ – Brale Sep 20 '19 at 12:49
  • $\begingroup$ @Brale_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy? $\endgroup$ – Sarvagya Gupta Sep 24 '19 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.