I'm a new student in reinforcement learning. Recently, I've been studying about different algorithms of RL. But I'm quite surprized that there are some algorithms which are named as "same" but they are working differently. For example, I tried running both this code from pytorch, and this code from CleanRL for deep Q learning in LunarLander-v2 gym environment. The main idea is to compute the temporal difference between $r+max_a Q'(s',a')$ and $Q(s,a)$ and minimize it (considering it as a loss for optimization in the neural network). Now if you look closely, both of the algorithms have some similar hyper parameters ($\gamma=0.99$, batch_size=128, memory_buffer=10000) and I tried changing the other hyperparameters to be the same (lr=10^{-4}, eps_start=0.9, tau=0.005). I also added in-place gradient clipping for the algorithm given in CleanRL (since it is not doing it). But I was surprized at the beginning that the CleanRL needs so many more episodes than the Pytorch one's to even get a reasonable average rewards (more than +200 for this environment). Later after doing some analysis, I found that CleanRL is not using the terminated episode anywhere. Hence I have the following question,

  1. I know in CleanRL they are running multiple environment's at once and trying to parallelize it, and using SyncVectorEnv function. But they have specifically mentioned in the code that assert args.num_envs == 1, "vectorized envs are not supported at the moment" which means that the number of parallel environment has to be 1, otherwise it will return an error. Since both algorithms are basically using one environment, why it is taking more number of episodes in cleanRL?

Please note that After running the Pytorch's code for only about 600 iterations, it was able to get desired average rewards (learned the model). But for CleanRL, I needed to run it for more than 10 million episodes !!! There are some more things I like to mention. For CleanRL, I changed the learning_starts hyperparameter from episode 10,000 to only 10, and train_frequency from 10 to 1. Because I realized it is not learning anything before episode 10,000. Running 600 episodes for Pytorch's algorithm takes nearly half an hour, while running 1 million episodes in CleanRL's algorithm takes about 20 minutes. Which algorithm should be better in this case? And what is exactly happening here?



You must log in to answer this question.