How to properly resume training of deep Q-learning network?

Question

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.

However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.

• Is this behaviour expected?
• If not, how do I properly resume training of a deep q-learning network?
• Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

Answer 1

Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.

Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.

From your description, I am assuming that you are monitoring the average reward per training episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.

Is this behavior expected?

I would expect it if you re-set $\epsilon$ to a high value.

If not, how do I properly resume training of a deep q-learning network?

I recommend that you look at the following things:

• Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.

• Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.

• Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.

You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.