It is fairly common in DQN to train a minibatch after every observation received after the replay memory has enough data (how much is enough is yet another parameter). This is not necessary, and it is fine to collect more data between training steps, the alogrithm is still DQN. The value higher than 1 for
train_freq here might be related to use of prioritised replay memory sampling - I have no real experience with that.
The update to target network generally needs to occur less frequently than training steps, it is intended to stabilise results numerically, so that over or under estimates of value functions do not result in runaway feedback.
The parameters choices will interact each other, most hyperparameters in machine learning do so unfortunately. Which makes searching for ideal values fiddly and time-consuming.
In this case it is safe to say that
train_freq is expected to be much lower than
target_update_freq, probably by at least an order of magnitude, and more usually 2 or 3 orders of magnitude. However, that's not quite the same as saying there is a strong relationship between choices for those two hyperparameters. The value of
batch_size is also relevant here, as it shows the rate that memory is being used (and re-used) by the training process.
The library you are using has these defaults:
batch_size::Int64 = 32
train_freq::Int64 = 4
target_update_freq::Int64 = 500
They seem like sane starting points. You are relatively free to change them as if they were independent, as there is no simple rule like "target_update_freq should be 125 times train_freq". As a very rough guide, you can expect that high values of
train_freq, low values of
batch_size and low values of
target_update_freq are likely to cause instability in the learning process, whilst going too far in the opposite direction may slow learning down. You might be able to set
train_freq to 1, but I am not completely certain about that either in combination with the prioritised replay memory sampling which seems to be the default in the library you are using.