# Should the network weights converge when training Deep Q networks?

I have two sets of data, one training and one test set. I use the train set to train the deep q network model variant. I also continuously evaluate the agent Q values obtained on the test set every 5000 epochs I find that the agent Q values on the test set do not converge and neither do the policies.

iteration $$x$$: Q values for the first 5 test data are [15.271439, 13.013742, 14.137051, 13.96463, 11.490129] with policies: [15, 0, 0, 0, 15]

iteration $$x+10000$$: Q values for the first 5 test data are [15.047309, 15.5233555, 16.786497, 16.100864, 13.066223] with policies: [0, 0, 0, 0, 15]

This means that the weights of the neural network are not converging. Although I can manually test each policy at each iteration and decide which of the policy performs best, I would like to know if correct training of the network would lead to weight convergence ?

Training loss plot:

You can see that the loss decreases over time however, there are occasional spikes in the loss which does not seem to go away.

• if you already have the data, why are you using rl? would this not be a supervised/unsupervised problem? – David Ireland Jun 9 '20 at 9:10
• It is not a supervised learning problem. The train set contains experiences and can be seen as the "replay buffer", just that the no new experiences can be collected. As I cannot obtain the "rewards" for the current policy in real time, I am evaluating the trained DQN on the "replay buffer" and analysing the Q scores that the neural network would output on unseen states (test data) for every few cycles of epochs – calveeen Jun 9 '20 at 9:49
• how do you generate the experience? will the experience you store not be from a really poor policy since you're not optimising at the same time? it sounds like you would have to gather experience, train the dqn a bit to improve the policy, then get more experience, otherwise you will just continue to get experience from a poor policy and the dqn could never get close to the optimal solution. – David Ireland Jun 9 '20 at 10:10
• so because I am trying to apply reinforcement learning techniques on clinical data, it would not be possible for me to gather new experiences and I would have to work with the current collected experiences demonstrated by the clinicans. – calveeen Jun 9 '20 at 10:12
• what type of clinical data is it? are you trying to figure out the best treatment group? – David Ireland Jun 9 '20 at 10:22