I am reading this paper and in algorithm 3 they describe an $n$-step Q-Learning algorithm. Below is the pseudo-code.
From this pseudo-code, it looks as though the final tuples that they would visit in don't get added to the memory buffer $M$. They define a sample size $T$, but also say in the paper that an episode terminates when $|S| = b$.
This leaves me two questions:
- Have I understood the episode termination correctly? It seems from the pseudocode they are just running an episode for $T$ time steps but also in the paper they have a definition for when an episode terminates, so I'm not sure why they would want to truncate the episode size.
- As I mentioned, it seems as though the final state $S_T$ that you would be in won't get added to the experience buffer as we only append the $(T-n)$th tuple. Why would you want to exclude the information you get from the final tuples you visit?