# Can the law of iterated expectation be used on the inner expectation of the DQN cost function described in the DQN paper

Is the expression for the DQN cost function, Equation (2) of the DQN paper

\begin{align}L_1 &= E_{\mu,\pi}\left[\left(y_i - q(s,a;\theta)\right)^2\right]\\ &=E_{\mu,\pi}\left[\left(E_{\mathcal{E}}[r + \gamma \max\limits_{a'}q(s',a';\theta^-)] - q(s,a;\theta)\right)^2\right] \end{align}

equivalent to this? (Substituting the expression for $$y_i$$ defined in the paragraph directly after, $$\mathcal{E}$$ represents the transition distribution governed by the environment, $$\pi$$ represents the behaviour policy and $$\mu$$ represents the stationary distribution of states)

$$L_2 = E_{\mu,\pi,\mathcal{E}}\left[\left(r + \gamma \max\limits_{a'}q(s',a';\theta^-) - q(s,a;\theta)\right)^2\right]$$

Can the law of iterated expectation be used to derive the second expression from the first, if not, is there another way to go about showing their equivalence IF they are equivalent.

It seems as though $$L_2$$ is used for sampling but I'm not sure how it's possible to get here from the original cost function $$L_1$$. If it is possible to use $$L_2$$ to sample I assume that means the two expressions must be equivalent. The second expression is used for sampling in the DQN paper here.

I do realise that the gradient for each function is the same and thus so is the $$n^{th}$$ derivative for some $$n\geq1$$ and since the curvature and optimas align I guess that also means they are the same function (minus some constant difference)?

$$\nabla_{\theta} L = E_{\mu,\pi,\mathcal{E}}\left[\left(r + \gamma \max\limits_{a'}q(s',a';\theta^-) - q(s,a;\theta)\right)\nabla q(s,a;\theta)\right]$$

Related Question

A related problem that concerns equivalence of sampling from $$L_1$$ and $$L_2$$. Is it possible to sample from a nested expectation that is squared as follows?

$$E[E[X|Y]^2] \approx \frac{1}{n}\sum X^2$$

Where $$X$$ is generated according to the marginalised distribution $$P(X)$$. I don't think it is true since $$E[X]^2 \neq E[X^2]$$ which should mean sampling from $$L_1$$ and $$L_2$$ are not equivalent.

• what is $\mathcal{E}$? Mar 4, 2021 at 16:32
• I don't see your first expression/formula anywhere in the DQN paper. Can you clarify this?
– nbro
Mar 5, 2021 at 11:37
• @nbro Equation (2) substitute y_i for the expression specified in the paragraph below Mar 5, 2021 at 11:39
• It's better you add this information to your post, i.e. how you get to your first expression given the expressions in the DQN paper.
– nbro
Mar 5, 2021 at 11:44
• Did you write the related question yourself or is it an actual fact. If it is an actual fact I would be very interested to know it's proof.
– user9947
Mar 5, 2021 at 14:04

\begin{align} E_{\mu,\pi}[E_{\mathcal{E}}[s'|s,a]^2] &= \sum_\limits{s,a}\left(\sum_\limits{s'}s'p(s'|s,a)\right)^2p(s,a) \\ &=\sum_\limits{s,a}\left(\sum_\limits{s_1'}s_1'p(s'_1|s,a)\cdot\sum_\limits{s_2'}s_2'p(s'_2|s,a)\right)p(s,a) \\ &= \sum_\limits{s,a,s_1',s_2'}s_1's_2'p(s_1'|s,a)p(s_2'|s,a)p(s,a) \\ &= E_{\mathcal{E}\sim (s_1',s_2'),\mu,\pi}[s_1'\cdot s_2'] \end{align}
This is the closest I could come up with, basically instead of the same sample squared, we pick two samples independently given the pair $$(s,a)$$.