I ran a deep q learning algorithm (DQN) for $x$ number of epochs and got policy $\pi_1$. I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script. However, when computing the actions on the same test set, i realised the actions were very different.

Is this supposed to be normal when training deep q networks or is there something that I am missing ?

I am using prioritised experience replay when training the model.

  • $\begingroup$ Have you measured the expected returns for the two different policies (separately from training runs)? If not, may be worth doing so and adding that detail to the question, as if the returns are similar it points to a different kind of answer to if they were really different $\endgroup$ – Neil Slater May 21 '20 at 8:52
  • $\begingroup$ Second clarifying question - are you calculating actions given arbitrary test states? Do those test states actually occur naturally when the agents behave optimally? $\endgroup$ – Neil Slater May 21 '20 at 8:53
  • $\begingroup$ not yet because it is not straight forward to measure the value of the learnt policy because i would have to use off policy reinforcement learning evaluation techniques like Weighted importance sampling to do so. $\endgroup$ – calveeen May 21 '20 at 8:55
  • $\begingroup$ regarding second question, I have trajectories of a behaviour policy, and i split them into training and test sets. After training the Q network only on the train set, I feed the test set into the Q network and obtain the actions via an argmax for each state $\endgroup$ – calveeen May 21 '20 at 8:58
  • $\begingroup$ @calveen: Is all this because you have one real environment, that your agents are not allowed to act in, and no way to accurately simulate it? $\endgroup$ – Neil Slater May 21 '20 at 9:42

Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?

Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):

  • Neural network weight initialisation
  • Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
  • Minibatch sampling from experience replay
  • Stochastic choices in the environment (does not apply in your case)
  • Other stochastic parts of training that may be in use, such as dropout regularisation

Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.

I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script.

This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:

  • Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.

  • Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.

You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.

Here are a couple of additional tests that may help:

  • Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.

  • Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.

If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?

Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.

You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.

  • $\begingroup$ Thanks Neil for the tips :-) I realised that it could be different because of prioritised replay sampling and initialisation of random normal weights. I wouldn't be able to believe if those random parameters affect the results :/ $\endgroup$ – calveeen May 21 '20 at 11:23
  • $\begingroup$ can prioritised experience replay be made deterministic ? because the sampling is based on probabilities ? $\endgroup$ – calveeen May 21 '20 at 11:33
  • $\begingroup$ @calveen: Any "random" process in machine learning can be made deterministic by setting the RNG seeds. That won't necessarily make things better for your end result though. A Q function that is consistently inaccurate has limited use $\endgroup$ – Neil Slater May 21 '20 at 11:34
  • $\begingroup$ Just to clarify again, for the 1st test, we compute the q value for the action chosen by the behaviour policy at some state for each run of the script and we see whether the q values for that particular action are similar ? For the second test, we use the q network from 1st run to compute the q function for the action chosen by the q network from the second run over some states and vice versa. The q values output by these 2 networks should be similar ? $\endgroup$ – calveeen May 21 '20 at 12:49
  • $\begingroup$ @calveen: Yes. Note those tests I am suggesting are not formal parts of Q-learning theory. They are just metrics that may help you decide whether you have a problem with stability of your learning algorithm. $\endgroup$ – Neil Slater May 21 '20 at 13:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.