3
$\begingroup$

I was working on an RL problem and I am confused at one specific point. We use replay memory so that the network learns about previous actions and how these actions lead to a success or a failure.

Now, to train the neural network, we use batches from this replay or experience memory. But here's my confusion.

Some places like this extract random (non-sequential) batches from the memory to train the neural network but Andrej Karpathy uses the sequential data to train the network.

Can someone tell me why there's the difference?

|improve this question
$\endgroup$
  • $\begingroup$ You linked 2 completely different algorithms, that is the reason why there is a difference. One is off-policy and other is on-policy algorithm $\endgroup$ – Brale_ Sep 20 '19 at 10:42
  • $\begingroup$ What do you mean? $\endgroup$ – Sarvagya Gupta Sep 20 '19 at 12:17
  • 1
    $\begingroup$ First link talks about DQN algorithm, second link is implementation of policy gradient algorithm, reinforcement learning isn't only a single algorithm there are multiple kinds of algorithms, you need to read up a bit on it $\endgroup$ – Brale_ Sep 20 '19 at 12:49
  • $\begingroup$ @Brale_, so correct me if I'm wrong. in DQN, the sequence doesn't matter because we have this Q table that gives an idea to the network what it should do to converge. But in PG, we don't have such a table or any reference hence we have to follow the sequence to see which set of actions led to an optimal policy? $\endgroup$ – Sarvagya Gupta Sep 24 '19 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.