While studying DQN, I rarely if ever see an explanation of what to fit off actions to.
Meaning, if the highest valued action of 3 choices if the first, when training, what do we do with the other two seeing as we don't have a target for them since they weren't taken?
I've seen some code that leaves the target for off actions as whatever the prediction returned, which feels a bit wrong to me as two or more similar behaving actions might never be differentiated well after random action selection dwindles.
I've also seen some implementations that set the target for all actions to zero and only adjust the target for the action taken. This would help regarding action differentiation long term, but it also puts more reliance on taking random actions for any unfamiliar states (I believe) as an off action might never be taken otherwise.