2
$\begingroup$

In a Deep Q-learning algorithm, we perform a batch training every train_freq and we update the parameters of the target network every target_update_freq. Are train_freq and target_update_freq necessary related, e.g., one should be always greater than the other, or must they be independently optimized depending on the problem?

EDIT Changed the name of batch_freq to train_freq.

$\endgroup$
3
  • $\begingroup$ It's more usual in my experience to run a moderately sized batch on every step. However, there is a lot of variation within the DQN method. Where are you seeing the batch_freq parameter? If you link the library or paper where this param is discussed I can probably figure out an answer $\endgroup$ – Neil Slater Jun 18 '20 at 7:09
  • $\begingroup$ If batch_freq is batch size then no. Neither in my experience nor in any paper I've read $\endgroup$ – mirror2image Jun 18 '20 at 7:16
  • $\begingroup$ @NeilSlater I think it's the standard implementation of DQN. The code is from github.com/JuliaPOMDP/DeepQLearning.jl/blob/master/src/…. Line github.com/JuliaPOMDP/DeepQLearning.jl/blob/… is doing the minibatch training every train_freq. Then, line github.com/JuliaPOMDP/DeepQLearning.jl/blob/… is doing the update every target_update_freq. $\endgroup$ – zdm Jun 18 '20 at 14:03
2
$\begingroup$

It is fairly common in DQN to train a minibatch after every observation received after the replay memory has enough data (how much is enough is yet another parameter). This is not necessary, and it is fine to collect more data between training steps, the alogrithm is still DQN. The value higher than 1 for train_freq here might be related to use of prioritised replay memory sampling - I have no real experience with that.

The update to target network generally needs to occur less frequently than training steps, it is intended to stabilise results numerically, so that over or under estimates of value functions do not result in runaway feedback.

The parameters choices will interact each other, most hyperparameters in machine learning do so unfortunately. Which makes searching for ideal values fiddly and time-consuming.

In this case it is safe to say that train_freq is expected to be much lower than target_update_freq, probably by at least an order of magnitude, and more usually 2 or 3 orders of magnitude. However, that's not quite the same as saying there is a strong relationship between choices for those two hyperparameters. The value of batch_size is also relevant here, as it shows the rate that memory is being used (and re-used) by the training process.

The library you are using has these defaults:

    batch_size::Int64 = 32
    train_freq::Int64 = 4
    target_update_freq::Int64 = 500

They seem like sane starting points. You are relatively free to change them as if they were independent, as there is no simple rule like "target_update_freq should be 125 times train_freq". As a very rough guide, you can expect that high values of train_freq, low values of batch_size and low values of target_update_freq are likely to cause instability in the learning process, whilst going too far in the opposite direction may slow learning down. You might be able to set train_freq to 1, but I am not completely certain about that either in combination with the prioritised replay memory sampling which seems to be the default in the library you are using.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.