DQN for Atari takes considerable training time. For example, the 2015 paper in Nature notes that algorithms are trained for 50 million frames or equivalently around 38 days of game experience in total. One reason is that DQN for image data typically uses a CNN, which is costly to train.

However, the main purpose of a CNN is to extract the image features. Note that the policy for DQN is represented by a CNN and an output layer equal to the number of discrete actions. Is it possible to use a pretrained DQN to accelerate the training process by fixing the weights of the underlying pretrained CNN, resetting the weights of the output layer, and then running another (possibly different) DQN algorithm to relearn the weights of the output layer? Both DQN algorithms would be run on the same underlying environment.

  • $\begingroup$ Could you please cite or explain why DQN for Atari takes 7+ days for training? Is it because you are working on a certain Atari environment or using a specific DQN architecture? $\endgroup$ – DeepQZero Jun 29 '20 at 15:58
  • 1
    $\begingroup$ Natures DQN. I use Dopamine”s implementation. What is your experience for Atari? $\endgroup$ – YoYO Man Jun 29 '20 at 15:59
  • $\begingroup$ What do you intend to pre-train DQN on? Another environment? $\endgroup$ – DeepQZero Jun 29 '20 at 16:06
  • 1
    $\begingroup$ Same environment. I am trying to use my own implementation of RL algorithm, by removing the last layer of a pretrained DQN. (The preceding layers are just CNN that does the image feature extraction) $\endgroup$ – YoYO Man Jun 29 '20 at 16:08
  • $\begingroup$ For a Bachelor's project, a friend and I did the same; and getting satisfactory results never took far more than 7 days or so. To get really good scores, yes. But the onset of training could be observed much earlier. But if you just want to pretrain a CNN, what you could do, is, for example, developing an autoencoder that you train on re-creating some atari-game frames. I guess you are working on the Gym environment? Then, sampling some training data for an autoencoder should not be a problem. Then, you could just take the encoder of the autoencoder as your CNN. $\endgroup$ – Daniel B. Jul 1 '20 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.