I have been getting into RL, and I have a DDQN model that learns how to play the super mario 1-1 world. Then, I tried using the code from the IQN paper to play this game (modified the DQN" part of the model to match the DDQN one, 3 conv layers then 2 fully connected).

Training with the same parameters, the IQN model takes longer to train, and reaches worse performance than the DDQN model does.

I have set the common parameters to be equal between them, and the only noticeable difference I've found is that IQN performs soft updates between target and local networks, while on DDQN I perform a hard copy every 1000 episodes.

Both are trained with a fixed low exploration rate (0.02). On three different runs, the results have been consistently better for simple DDQN.

Any idea on why this could be happening? Could this difference be justified just by better parameter tuning on the DDQN model? Parameter count?

Thanks in advance

The parameters used are:

Parameter Value
Gamma 0.9
Epsilon 0.02
Lr 0.00025
Batch Size 16
Tau (IQN) 0.01
Update* 1

*Update every n frames.

Reward graphs (smoothed using ewm):



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.