I'm getting started with DRL and have trouble distinguishing TD(0), MC, and GAE; and which scenarios one's better than others. Here is what I understand so far:
TD(0): increment learning, can learn after each step instead of waiting for the episode to end. Update bases on one reward, then low variance. However, high bias.
MC: Learn after each episode, the calculation of return is correct. However, its drawback is high variance. And you have to make decisions in the whole episode without an update of parameters.
GAE: combine returns in all steps, get a better trade-off between variance and bias. However, still has to wait until the end of the episode for an update.
I have some questions as follows:
Is variance and bias about the return of each episode? What are their effects on the outcomes (convergence speed of training process, performance of the model)?
Is increment learning important? The ability to correct behaviors after each step may improve convergence speed. However, it can lead to unstable learning ( if I understand correctly, this is why the Target model in Double DQL only updates its parameters for each k mini-batches ). Which scenarios should I use TD(0) or GAE?
Concretely, in my case, I run parallelly a batch with 12 environments, each with 1000 steps. If I use GAE, I make 12000 decisions for each update. All losses of the model are summed up and calculate gradients, after that, I clip gradients to 2.0. Is that too expensive to learn the correct direction? Should I consider using TD(0) here?