3
$\begingroup$

I've already read the original paper about double DQN but I do not find a clear and practical explanation of how the target $y$ is computed, so here's how I interpreted the method (let's say I have 3 possible actions (1,2,3)):

  1. For each experience $e_{j}=(s_{j},a_{j},r_{j},s_{j+1})$ of the mini-batch (consider an experience where $a_{j}=2$) I compute the output through the main network in the state $s_{j+1}$, so I obtain 3 values.

  2. I look which of the three is the highest so: $a^*=arg\max_{a}Q(s_{j+1},a)$, let's say $a^*=1$

  3. I use the target network to compute the value in $a^*=1$ , so $Q_{target}(s_{j+1},1)$

  4. I use the value at point 3 to substitute the value in the target vector associeted with the known action $a_{j}=2$, so: $Q_{target}(s_{j+1},2)\leftarrow r_{j}+\gamma Q_{target}(s_{j+1},1)$, while $Q_{target}(s_{j+1},1)$ and $Q_{target}(s_{j+1},3)$, which complete the target vector $y$, remain the same.

Is there anything wrong?

$\endgroup$
1
$\begingroup$

$$Y_{t}^{\text {DoubleDQN }} \equiv R_{t+1}+\gamma Q\left(S_{t+1}, \underset{a}{\operatorname{argmax}} Q\left(S_{t+1}, a ; \boldsymbol{\theta}_{t}\right), \boldsymbol{\theta}_{t}^{-}\right)$$

The only difference between the "original" DQN and this one is that you use your $Q_\text{est}$ with the next state to get your action (by choosing the action with the highest Q).

Afterward, you just figure out what the target $Q$ is given that action, by selecting the $Q$ belonging to that action from the target_network (instead of using the argmax a directly on the target Q network).

About the formula

  • $\theta_{t}^{-}$ above it means frozen weights, so it represents the target Q network.

  • the other $\theta_{t}$ represents the "learnable weights" so the estimate Q network.

$\endgroup$
  • $\begingroup$ Yes of course, but the question is quite different, maybe I've not explained well. Try look at this question: stats.stackexchange.com/questions/306921/… $\endgroup$ – aandre_90 Aug 16 '20 at 9:17
  • $\begingroup$ In other words, "to construct a target vector" do I have to use the target network or the main network? $\endgroup$ – aandre_90 Aug 16 '20 at 9:21
  • $\begingroup$ Your Q target vector that you want to converge to is the same as your output vector except for the Q at the index of the chosen action, that one is the Y_doubleDQN value described in the paper. $\endgroup$ – hal9000 Aug 16 '20 at 11:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.