When training a DQN, how should we update the value of actions that were not taken?

Question

Let's say that we have three actions. The highest-valued action of the three choices is the first. When training the DQN, what do we do with the other two, as we don't have a target for them, since they weren't taken?

I've seen some code that leaves the target for off actions as whatever the prediction returned, which feels a bit wrong to me as two or more similar behaving actions might never be differentiated well after random action selection dwindles.

I've also seen some implementations that set the target for all actions to zero and only adjust the target for the action taken. This would help regarding action differentiation long term, but it also puts more reliance on taking random actions for any unfamiliar states (I believe) as an off action might never be taken otherwise.

Answer 1

The loss function for DQN algorithm is \begin{equation} L(\theta_i) = \mathbb E_{s, a, r, s'} [(y - Q(s, a;\theta_i))^2] \end{equation} Like you said, we only take one action per timestep. We can only shift weights of the network that had the effect in calculating action value $Q(s, a)$ for that particular action that we took. For that action, variable $y$ would have value \begin{equation} y = r + \gamma \max_{a'} Q(s', a', \theta^-_i) \end{equation} and we would have standard form of DQN loss. We calculate the gradient of that loss with respect to network parameters $\theta$, backprop it and slightly shift weights so to more accurately estimate $Q(s, a)$ for that specific state-action pair.

We never took other actions. Since we never took them we didn't get reward $r$ and we can't estimate $\max_{a'} Q(s', a', \theta^-_i)$ for those other actions. We don't want to change weights that had effect on calculating actions values for those other actions because we have no way of estimating how accurate they were. Standard approach is to set \begin{equation} y = Q(s, a;\theta_i) \end{equation} this way the loss for those other actions would be $0$ and it would result in not changing the weights that had influence in calculating their action values.

Setting target $y$ to $0$ for all other actions would mean that we want $Q(s, a)$ for them all to slightly shift to $0$. That would not be correct since we have no way of knowing their true value. I think you misinterpreted that part in the implementations.