Why td_loss is calculated from (td_targets against q_values)?

Why I am lost is because:

  1. q_values is just the probability of action. It does not have a reward and discount.
  2. td_targets does have rewards + discounts * next_q_values. Somemore next_q_values is next state.

How both td_targets and q_values can minus (or Huber or MSE) to get lost work?

td_error = valid_mask * (td_targets - q_values)
td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)

td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)


First of all, DQN is off-policy learning. That means, you are following the behavior policy(epsilon greedy policy) but still learning about the optimal policy or target policy(greedy policy). Td_target in DQN is the estimation of our current state's optimal action-value function independent of the policy we are following (since we are picking next state's action-value from target policy) and q_values(as you referred) is what you get following the behavior policy. While using this kind of update, you are improving both the behavior policy and the target policy.

  • $\begingroup$ Td_target is the estimation of our current state's but it q value is base on next state. $\endgroup$ – robot tech Apr 26 '20 at 12:26
  • $\begingroup$ In DQN, the td_target is the optimal q value and you are subtracting it from your estimated q value. $\endgroup$ – Swakshar Deb Apr 26 '20 at 12:49
  • $\begingroup$ As i mentioned above, td_targets is tf.stop_gradient(rewards + discounts * next_q_values). It is not just simple next_q_values. $\endgroup$ – robot tech Apr 26 '20 at 12:53
  • $\begingroup$ In DQN, td_targets = rewards + discount*(max q value of next state). The max term comes from the target policy. If the explanation did not satisfy you, can you please explain to me your question in more detail(without adding the code section)? $\endgroup$ – Swakshar Deb Apr 26 '20 at 13:25
  • $\begingroup$ I am just do not understand how come it working. Maybe i need more this kind of lost solution on other senario. Just for appreciation and have intuition. $\endgroup$ – robot tech Apr 27 '20 at 12:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.