So I am trying to understand and make a DQN. But I didn't understand a part. So basically state's Q values computed with the network and the target Q values will also compute with a target network according to the next state. Then the results will pass into the Bellman equation that is (R(t+1) + Q_max(s(t+1),a'). But this taking the maximum thing makes all actions' Q values same, right? And target's Q values being same means the network try to make predictions for different actions (in other words network's outputs) very similar. Am I missing somewhere or is there something I didn't know?
2
-
1$\begingroup$ please consider checking the structure/spelling of the question, it's very hard to get what's your point $\endgroup$– AlbertoAug 6 at 19:04
-
$\begingroup$ Well, you can ask the same question without the target network. What happens in tabular Q-learning? There's a max there too. $\endgroup$– nbroAug 7 at 11:43
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
First, the problem you're raising also applies to tabular Q-learning or DQN without the target network because the max operation exists in those algorithms too.
Second, note that the Q-network should be initialised randomly, so that means that the initial $Q(s, a)$ should be different for all states and actions pairs (at least at the beginning).
Let's ignore the target network then.
Now, you should look at algorithm 1 in the DQN paper
You're right the max operation selects only one value. However, as you can see from the algorithm, we don't use this target value to update all state-action pairs, but just one, the one at time-step $t$, denoted by $(s_t, a_t)$ in tabular Q-learning, and $(\phi_t, a_t)$ in DQN. The target would be denoted by $y_t$ in DQN.
Note also that the target is not just the max, but also the reward. The reward you observe may be different for each state-action. So, no, the target value is not always the same, even if you keep the target network frozen forever, which is not usually the case - we update the target network every $C$ steps.
-
$\begingroup$ I know the reward(prediction of the target network) will change in time. I understood mostly you said thanks but just to make sure one more question: So, the prediction network's "target output" for the third output will be the value of the third output of the target network even if it isn't the maximum Q value between the predictions, right? $\endgroup$– EgeAug 7 at 12:13
-
$\begingroup$ The reward and the prediction of the target network are not the same thing. The target network estimates the return. The return is not the same thing as the reward. The return is the (cumulative/discounted) sum of rewards. The reward that I meant was what you denoted by $R(t+1)$. In the paper's algorithm, it's denoted by $r_t$. I am not sure what you mean by "prediction network". In any case, the target value is calculated using $r_t$, which depends on the action that you take while exploring (see the algorithm), and the maximum value at the state $s_t$. $\endgroup$– nbroAug 7 at 13:14
-
$\begingroup$ In the same state $s_t$, the maximum Q-value is always the same if the Q-network does not change. However, it might be different across states, of course. The reward $r_t$ in the same state $s_t$ may also be different. This depends on the reward function (which is not the reward, return or Q-network - it's a function that also defines the MDP, problem, and gives you the reward) because you may end up in the state $s_t$ because of different events. $\endgroup$– nbroAug 7 at 13:15
-
$\begingroup$ Yes sorry for the stupid mixing the reward and target Q value, when I am writing it, in my mind it was Q value but I wrote it as reward. Silly mistake. I am asking again and again sorry but I want to understand it well. To be more clear, in my DQN I am using MSE loss function. All I am asking is when calculating the MSE's derivative for the gradient decant, I am subtracting the predictions from target values for all output values(O1-o1, O2-o2, O3-o3...) and should I do that like this also in DQN. I am confused because at DQN loss is calculated according to max of the Q values beside the normal $\endgroup$– EgeAug 7 at 16:24
-
$\begingroup$ If you're doing some kind of batch GD, you should calculate the maximum value and reward, so the target, for each different $(s_t, a_t)$. They might not be the same. But, tbh, I'm not familiar with the typical DQN implementations. $\endgroup$– nbroAug 7 at 21:30