0
$\begingroup$

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it normal that Q values for the non-terminal states have higher values than those of the terminal states?

The reason why I want to know whether Q values learnt are appropriate is because I want to apply the doubly robust estimator for off-policy value evaluation. Using doubly robust requires a good Q value estimate to be learnt for each state.

Improve this question
$\endgroup$
5
  • $\begingroup$ To answer the question "is it normal that Q values for the non terminal states have higher values than those of the terminal states ?", you should ask yourself what the value function represents. What is the definition of the value function? After that, you should be able to answer your question ;) $\endgroup$ – nbro♦ Jun 14 '20 at 13:02
  • $\begingroup$ I believe the answer is yes the Q values for non terminal states can be higher, an example would be possible if the rewards from the start state to the terminal state is positive. However, the Q values are insanely high compared to the rewards. For the tabular learning it is easier to visualise and understand the Q values. For the continuous state space however, i am not sure how to wrap my head around that. $\endgroup$ – calveeen Jun 14 '20 at 13:34
  • $\begingroup$ My rewards are of magnitude 10^-2 - 10^-1 range. but my Q values are of magnitude 10^1 - 10^2. Hence I believe something is off with my training.. $\endgroup$ – calveeen Jun 14 '20 at 13:36
  • $\begingroup$ Well, maybe you should tell us about the discount factor, learning rate and how many steps and episodes, etc. That could help people to understand better if there's something wrong. $\endgroup$ – nbro♦ Jun 14 '20 at 13:43
  • $\begingroup$ I will try to look more into the problem first.. $\endgroup$ – calveeen Jun 14 '20 at 15:02
1
$\begingroup$

DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From my experience, high learning rate usually cause over approximated Q, or mistakes made in the code. Best way to check is to see plot of Q function when running on environment to see if it makes any sense. (e.g. It should go lower for bad states and vice versa)

Improve this answer
$\endgroup$
4
  • $\begingroup$ What does "over-approximation on Q function" mean? Can you provide at least one source (i.e. a research paper) that supports that claim? $\endgroup$ – nbro♦ Jun 15 '20 at 10:27
  • $\begingroup$ Double Q-learning (Hado van Hasselt) $\endgroup$ – daidew Jun 15 '20 at 15:06
  • $\begingroup$ Please, just edit your answer to provide the link to the paper. $\endgroup$ – nbro♦ Jun 15 '20 at 15:35
  • $\begingroup$ You’re welcome ;) $\endgroup$ – daidew Jun 16 '20 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.