2
$\begingroup$

I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code.

pseudo-code for n-step q-learning

From this pseudo-code, it looks as though the final tuples that they would visit in don't get added to the memory buffer $M$. They define a sample size $T$, but also say in the paper that an episode terminates when $|S| = b$.

This leaves me two questions:

  1. Have I understood the episode termination correctly? It seems from the pseudocode they are just running an episode for $T$ time steps but also in the paper they have a definition for when an episode terminates, so I'm not sure why they would want to truncate the episode size.
  2. As I mentioned, it seems as though the final state $S_T$ that you would be in won't get added to the experience buffer as we only append the $(T-n)$th tuple. Why would you want to exclude the information you get from the final tuples you visit?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.