Can deep reinforcement learning algorithms be deterministic in their reproducibility in results?
Yes, but only if you control all places in the code where stochastic methods are used (typically by seeding the affected RNGs):
- Neural network weight initialisation
- Action choice for $\epsilon$-greedy or other behaviour policy (does not apply in your case, because you work exclusively from experience replay)
- Minibatch sampling from experience replay
- Stochastic choices in the environment (does not apply in your case)
- Other stochastic parts of training that may be in use, such as dropout regularisation
Controlling all these should make your training process deterministic and repeatable. It won't necessarily make it correct.
I reran the same script for the same $x$ number of epochs and got policy $\pi_2$. I expected $\pi_1 $ and $\pi_2$ to be similar because i ran the same script.
This is subtly different. It seems you hoped that convergence of the algorithm would mean you got to the same approximately optimal policy. In principle this is possible, because Q-learning should find a deterministic policy. However, there are some details to bear in mind:
Many environments support multiple equivalent optimal policies. A simple grid world can have multiple equivalent paths from start to goal states. A Q-learning with approximation function will slightly prefer one or other path, resulting in very different, but still optimal, policies.
Q-learning with approximation can go wrong and learn incorrectly. The usual checks and balances against this are running large numbers of simulations and testing.
You don't have great options here, from your comments you are training purely offline from historic data. Your one sanity check - do I get the same policy if I re-try - has shown inconsistency. However, it doesn't necessarily mean you have a problem, perhaps the two policies are equivalent.
Here are a couple of additional tests that may help:
Instead of looking at the maximising action choice in the test data, look at how each Q function scores the behaviour policy action choice. If the scores are close (by some measure such as MSE), then the two Q-learners are basically agreeing and are more likely to have equivalent but different policies, as opposed to radically different end results.
Have each Q network score the other's Q function action choice over an arbitrary (but realistic) set of states. If the values are similar to each other, then again this points to successful convergence given the training data, but with different outcomes due to small details.
If either of these checks shows the networks are radically different, then you have a problem. Which run, if any, has found a viable policy, and which has failed?
Even if the checks agree, it is circumstantial evidence that the Q learning process is stable, not proof that you have an agent that is better than the prevailing behaviour policy in your real world system.
You won't know if the agent is truly better, unless you can find a more independent way to assess the agent.