I have two questions

  1. When we use our network to approximate our Q values, is the Q target a single value?

  2. During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?


When we use our network to approximate our Q values,is the Q target a single value?

Yes, the target Q value is a single value if you are just updating a single training example. The loss function of a vanilla DQN for a single experience tuple $(s_t,a_t,r_t,s_{t+1})$ is calculated as $$L(\theta) = [r_t + \gamma \,max\,Q_{a_{t+1}}(s_{t+1},a_{t+1};\theta) - Q(s_t,a_t;\theta)]^2$$ where $r_t + \gamma \,max\,Q(s_{t+1},a_{t+1};\theta)$ is the target Q value. However, when using mini-batch gradient descent, you would have to compute multiple target Q values equivalent to the batch size

During backpropagation, when the weights are updated, does it automatically update the Q values, shouldn’t the state be passed in the network again to update it?

During backpropagation of the loss function, the weights $\theta$ are automatically updated. You do not need to pass in the state again. Because in the first place, you would have computed $Q(s_t,a_t;\theta)$ by passing in the state as input to the neural network. That is how backpropagation works for Deep Q networks.

Training for the DQN is as follows:

  1. Collect experience tuples of $(s_t,a_t,r_t,s_{t+1})$ and store them in a replay buffer.
  2. Sample mini-batch of experiences from the replay buffer.
  3. From these sampled batch of experiences, compute $Q(s_{t+1}.a_{t+1};\theta)$ by passing $s_{t+1}$ into the network and take the Q value with the maximum values
  4. Compute $Q(s_t,a_t;\theta)$ by passing $s_t$ into the network.
  5. Compute the Loss for this experience and propagate the loss back to the network, hence updating the weights.

Also, since the weights have changed after backpropagation, the Q values for the same state would also be updated if you pass the same state in to the network again. Check out this paper as it explains how Deep Q Network works.

  • 1
    $\begingroup$ @Chukwudi You are very confused. I suggest that you first learn about tabular Q-learning before getting into deep Q-learning! The neural network $M_{\theta_i}$ represents your Q function at time step $i$. If you update the parameters of this neural network, you also change your Q function. So, after an update of the parameters of this network that represents the Q function, then $M_{\theta_{i+1}}(s)$ will (probably) be different than $M_{\theta_i}(s)$, where $\theta_i$ are the parameters of the net before the update. $\endgroup$ – nbro Jun 12 '20 at 22:47
  • $\begingroup$ I understand the table, I just realized that it’s a function approximation, so anytime the weights change the Q clause also change to move closer to the respective Q state action pair values , and we meet the weights of our target network constant to introduce stability, David Ireland explained it in detail $\endgroup$ – Chukwudi Jun 13 '20 at 10:35

What the network represents can be a little confusing - in tabular Q-learning you have a $Q$ function that you pass a state $s$ and action $a$ into and receive a scalar value. In the Human Level Control paper where the DQN gained its popularity, the network is a little different to the tabular function. You pass into the network your current state and it outputs values for every action in your action space. You then choose the action which has the highest value.

The way we then train it is as follows - first we get our scalar value which corresponds to the target $r + \gamma max_{a'} Q(s',a')$; I will denote this as $y$ for short. As our network outputs a vector of dimension $|\mathcal{A}|$ where $\mathcal{A}$ is our action space, we get our prediction for the current state which is a vector of the dimension I just mentioned. We will now assume our action space has two possible actions and say that for state $s$ our network outputs the vector $[\hat{y}_1,\hat{y}_2]$. Assume that when we calculated $y$ earlier we took the first action, so we only want to update our prediction of $\hat{y}_1$. To do this, if we were working in e.g. PyTorch I would put into the loss function, which is mean squared error,

input = $[\hat{y}_1,\hat{y}_2$], target = $[y, \hat{y}_2]$

so essentially all we change is the position in the vector corresponding to the action we took with the Q-target we observed. The weights will thus be updated accordingly to move more into the direction of producing this output for state $s$.

So to explicitly answer your first question, the Q-target is a scalar but the target we pass into the network is a vector.

  • 1
    $\begingroup$ I think you are misunderstanding what happens in Q-learning. The network is a function of the state. Imagine a normal neural network that is a function of x, when you update the weights the next time you feed the same x in it will (probably) give you a different value. There’s not really anything else that can be said unless you explain further what you don’t understand. $\endgroup$ – David Ireland Jun 12 '20 at 22:18
  • $\begingroup$ Ohh I think I get it, it’s a function approximation, so as long as the weights change , the next time the state is fed the Q values also change, it doesn’t matter about changing immediately $\endgroup$ – Chukwudi Jun 13 '20 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.