td_loss is calculated from (
Why I am lost is because:
q_valuesis just the probability of action. It does not have a reward and discount.
td_targetsdoes have rewards + discounts * next_q_values. Somemore next_q_values is next state.
q_values can minus (or Huber or MSE) to get lost work?
td_error = valid_mask * (td_targets - q_values) td_loss = valid_mask * td_errors_loss_fn(td_targets, q_values)