When we use a neural network to approximate the Q values, is the Q target a single value?

Question

I have two questions

1. When we use our network to approximate our Q values, is the Q target a single value?

2. During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?

Answer 1

When we use our network to approximate our Q values,is the Q target a single value?

Yes, the target Q value is a single value if you are just updating a single training example. The loss function of a vanilla DQN for a single experience tuple $(s_t,a_t,r_t,s_{t+1})$ is calculated as $$L(\theta) = [r_t + \gamma \,max\,Q_{a_{t+1}}(s_{t+1},a_{t+1};\theta) - Q(s_t,a_t;\theta)]^2$$ where $r_t + \gamma \,max\,Q(s_{t+1},a_{t+1};\theta)$ is the target Q value. However, when using mini-batch gradient descent, you would have to compute multiple target Q values equivalent to the batch size

During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?

During backpropagation of the loss function, the weights $\theta$ are automatically updated. You do not need to pass in the state again. Because in the first place, you would have computed $Q(s_t,a_t;\theta)$ by passing in the state as input to the neural network. That is how backpropagation works for Deep Q networks.

Training for the DQN is as follows:

1. Collect experience tuples of $(s_t,a_t,r_t,s_{t+1})$ and store them in a replay buffer.
2. Sample mini-batch of experiences from the replay buffer.
3. From these sampled batch of experiences, compute $Q(s_{t+1}.a_{t+1};\theta)$ by passing $s_{t+1}$ into the network and take the Q value with the maximum values
4. Compute $Q(s_t,a_t;\theta)$ by passing $s_t$ into the network.
5. Compute the Loss for this experience and propagate the loss back to the network, hence updating the weights.

Also, since the weights have changed after backpropagation, the Q values for the same state would also be updated if you pass the same state in to the network again. Check out this paper as it explains how Deep Q Network works.

Answer 2

What the network represents can be a little confusing - in tabular Q-learning you have a $Q$ function that you pass a state $s$ and action $a$ into and receive a scalar value. In the Human Level Control paper where the DQN gained its popularity, the network is a little different to the tabular function. You pass into the network your current state and it outputs values for every action in your action space. You then choose the action which has the highest value.

The way we then train it is as follows - first we get our scalar value which corresponds to the target $r + \gamma max_{a'} Q(s',a')$; I will denote this as $y$ for short. As our network outputs a vector of dimension $|\mathcal{A}|$ where $\mathcal{A}$ is our action space, we get our prediction for the current state which is a vector of the dimension I just mentioned. We will now assume our action space has two possible actions and say that for state $s$ our network outputs the vector $[\hat{y}_1,\hat{y}_2]$. Assume that when we calculated $y$ earlier we took the first action, so we only want to update our prediction of $\hat{y}_1$. To do this, if we were working in e.g. PyTorch I would put into the loss function, which is mean squared error,

input = $[\hat{y}_1,\hat{y}_2$], target = $[y, \hat{y}_2]$

so essentially all we change is the position in the vector corresponding to the action we took with the Q-target we observed. The weights will thus be updated accordingly to move more into the direction of producing this output for state $s$.

So to explicitly answer your first question, the Q-target is a scalar but the target we pass into the network is a vector.