Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?
The reason why I want to know whether Q values learnt are appropriate is because I want to apply the doubly robust estimator for off-policy value evaluation. Using doubly robust requires a good Q value estimate to be learnt for each state.