1
$\begingroup$

Is the expression for the DQN cost function, Equation (2) of the DQN paper

$$\begin{align}L_1 &= E_{\mu,\pi}\left[\left(y_i - q(s,a;\theta)\right)^2\right]\\ &=E_{\mu,\pi}\left[\left(E_{\mathcal{E}}[r + \gamma \max\limits_{a'}q(s',a';\theta^-)] - q(s,a;\theta)\right)^2\right] \end{align}$$

equivalent to this? (Substituting the expression for $y_i$ defined in the paragraph directly after, $\mathcal{E}$ represents the transition distribution governed by the environment, $\pi$ represents the behaviour policy and $\mu$ represents the stationary distribution of states)

$$L_2 = E_{\mu,\pi,\mathcal{E}}\left[\left(r + \gamma \max\limits_{a'}q(s',a';\theta^-) - q(s,a;\theta)\right)^2\right]$$

Can the law of iterated expectation be used to derive the second expression from the first, if not, is there another way to go about showing their equivalence IF they are equivalent.

It seems as though $L_2$ is used for sampling but I'm not sure how it's possible to get here from the original cost function $L_1$. If it is possible to use $L_2$ to sample I assume that means the two expressions must be equivalent. The second expression is used for sampling in the DQN paper here.

I do realise that the gradient for each function is the same and thus so is the $n^{th}$ derivative for some $n\geq1$ and since the curvature and optimas align I guess that also means they are the same function (minus some constant difference)?

$$\nabla_{\theta} L = E_{\mu,\pi,\mathcal{E}}\left[\left(r + \gamma \max\limits_{a'}q(s',a';\theta^-) - q(s,a;\theta)\right)\nabla q(s,a;\theta)\right]$$


Related Question

A related problem that concerns equivalence of sampling from $L_1$ and $L_2$. Is it possible to sample from a nested expectation that is squared as follows?

$$E[E[X|Y]^2] \approx \frac{1}{n}\sum X^2$$

Where $X$ is generated according to the marginalised distribution $P(X)$. I don't think it is true since $E[X]^2 \neq E[X^2]$ which should mean sampling from $L_1$ and $L_2$ are not equivalent.

$\endgroup$
6
  • 1
    $\begingroup$ what is $\mathcal{E}$? $\endgroup$ Mar 4 at 16:32
  • $\begingroup$ I don't see your first expression/formula anywhere in the DQN paper. Can you clarify this? $\endgroup$
    – nbro
    Mar 5 at 11:37
  • $\begingroup$ @nbro Equation (2) substitute y_i for the expression specified in the paragraph below $\endgroup$
    – quest ions
    Mar 5 at 11:39
  • $\begingroup$ It's better you add this information to your post, i.e. how you get to your first expression given the expressions in the DQN paper. $\endgroup$
    – nbro
    Mar 5 at 11:44
  • $\begingroup$ Did you write the related question yourself or is it an actual fact. If it is an actual fact I would be very interested to know it's proof. $\endgroup$
    – user9947
    Mar 5 at 14:04
0
$\begingroup$

Using the related question and simplifying notation

$$\begin{align} E_{\mu,\pi}[E_{\mathcal{E}}[s'|s,a]^2] &= \sum_\limits{s,a}\left(\sum_\limits{s'}s'p(s'|s,a)\right)^2p(s,a) \\ &=\sum_\limits{s,a}\left(\sum_\limits{s_1'}s_1'p(s'_1|s,a)\cdot\sum_\limits{s_2'}s_2'p(s'_2|s,a)\right)p(s,a) \\ &= \sum_\limits{s,a,s_1',s_2'}s_1's_2'p(s_1'|s,a)p(s_2'|s,a)p(s,a) \\ &= E_{\mathcal{E}\sim (s_1',s_2'),\mu,\pi}[s_1'\cdot s_2'] \end{align}$$

This is the closest I could come up with, basically instead of the same sample squared, we pick two samples independently given the pair $(s,a)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.