1
$\begingroup$

I'm currently training a deep q-learning network. Due to resource limitations, I am not able to train the model to the desired performance in one go. So what I'm doing now is training the model for a certain number of episodes and save the resultant model. Later, I will load up the previously saved model and resume training.

However, what I'm noticing is that when training resumes, the average rewards goes back to very low again, compared to what it achieved at the end of the previous training session. What I'm currently doing is to load up the previously saved model into the prediction and target models, and I keep all hyperparameters unchanged.

  • Is this behaviour expected?
  • If not, how do I properly resume training of a deep q-learning network?
  • Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?
$\endgroup$
0
1
$\begingroup$

Do I start off with the epsilon value at the end of the previous session, currently I reinitialize that as well?

You should probably re-start with $\epsilon$ at the value you left off at. Using high values of epsilon may cause the neural network to forget some of what it learned from close-to-optimal policies in favour of learning possibly useless values of states and actions that are not important to a more highly-trained agent.

Also, you should either save, or wait to refill experience replay memory before restarting training updates to the neural network. Working from a small memory may also cause the neural network to overfit to specific samples and generalise less well - at least temporarily until the memory fills up again.

From your description, I am assuming that you are monitoring the average reward per training episode. This is metric that is easy to collect, but that has a problem in off-policy RL. The problem is that you are seeing the results from your behaviour policy, not your learned policy. When using $\epsilon$-greedy in Q learning, with a high value of epsilon, then the behaviour policy is likely to perform badly.

Is this behavior expected?

I would expect it if you re-set $\epsilon$ to a high value.

If not, how do I properly resume training of a deep q-learning network?

I recommend that you look at the following things:

  • Re-start with $\epsilon$ at or close to where you left off. Perhaps allow starting epsilon to be passed in to the script as an argument.

  • Measure performance of the greedy policy at checkpoints - e.g. run 100 test episodes at the end of every 500 training episodes - and use plots of that to decide how well your agent is performing. You can still plot training performance, but it should not be your guide to how well the agent is learning.

  • Save experience replay so far at checkpoints too, and reload it on re-start. Alternatively, allow for experience replay to fill significantly before allowing update steps in the training loop.

You could consider these in priority order. Note that just the first one may appear to "fix" your problem, but that is because you are not yet properly measuring your agent's performance.

$\endgroup$
2
  • $\begingroup$ Thank you for your suggestions and the detailed explanation. As you have suggested, re-start at previous epsilon has helped solved the problem. Regarding the performance measurement, would it be correct to say that as epsilon becomes quite small, the reward obtained in training episode should be quite close to test result? $\endgroup$ – Tianxun Zhou Apr 13 at 4:43
  • $\begingroup$ @TianxunZhou Yes, the expected reward for $\epsilon$-greedy policy will become close to expected reward for greedy policy as $\epsilon$ becomes close to zero. It is not always clear how close to zero you need to get before that is true - that depends on the environment. $\endgroup$ – Neil Slater Apr 13 at 6:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.