I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make the conclusion that because the Q value for the target policy, $Q(s,\pi_e(s))$ is higher than the Q values for the behaviour policy, $Q(s,\pi_b(s))$ at all states encountered, where $\pi_e$ is the policy output by deep Q-learning and $\pi_b$ is the behaviour policy, then this target policy has better performance than the behaviour policy?
I know the proper way is to run the policy and do an empirical comparison of some sort. However, that is not possible in my case.