Why is the equation $\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon)$ true?

Question

In the book An Introduction to Statistical Learning, the authors claim (equation 2.3, p. 19, chapter 2)

$$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon) \label{0}\tag{0},$$

where

• $Y = f(X) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$ and $f$ is the unknown function we want to estimate
• $\hat{Y} = \hat{f}(X)$ is the output of our estimate of $f$, i.e. $\hat{f} \approx f$

They claim that this is easy to prove, but this may not be easy to prove for everyone. So, why is equation \ref{0} true?

Answer 1

Let's say we have $a$ - constant and $\epsilon \sim \mathcal{N}(0,\sigma)$, then: $$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$ Expectations of constants are just the constants: $\mathbb{E}[a] = a$ and $\mathbb{E}[a^2] = a^2$
The mean of $\epsilon$ is zero $\mathbb{E}[\epsilon] = 0$. And the expectation of $\epsilon^2$ is its variance: $$ \mathop{\mathrm{Var}}(\epsilon) = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$

Substituting, we get an expression for the original expectation:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{*}$$

Getting to the expectation in the book, we first substitute the values for $Y$ and $\hat{Y}$: $$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$

In the book it is assumed that $X$, $f$ and $\hat{f}$ are constant. So we can use the expression (*) with the constant being $a = f(X) - \hat{f}(X)$:

$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$

Answer 2

Let me try to show this. The only (non-constant) random variable here is $\epsilon$, while $f(X)$ and $\hat{Y} = \hat{f}(X)$ are constant random variables (so their expectations is equal to their only value).

So, we start with the following expression.

\begin{align} \mathbb{E} \left[ (Y - \hat{Y})^2 \right] \tag{1}\label{1} \end{align}

Now, we just apply the distributive property, so \ref{1} becomes

\begin{align} \mathbb{E} \left[ Y^2 - 2Y \hat{Y} + \hat{Y}^2 \right] \tag{2}\label{2} \end{align}

Given the linearity of the expectation, we can write \ref{2} as follows

\begin{align} \mathbb{E} \left[ Y^2 \right] - \mathbb{E} \left[ 2Y \hat{Y} \right] + \mathbb{E} \left[\hat{Y}^2 \right] \tag{3}\label{3} \end{align}

Given that $\hat{Y} = \hat{f}(X)$ is a constant and that we can take constants out of the expectations, we have

\begin{align} \mathbb{E} \left[ Y^2 \right] - 2 \hat{Y} \mathbb{E} \left[ Y \right] + \hat{Y}^2 \tag{4}\label{4} \end{align}

Now, let's replace $Y$ with $f(X) + \epsilon$, to obtain

\begin{align} \mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \mathbb{E} \left[ f(X) + \epsilon \right] + \hat{Y}^2 \tag{5}\label{5} \end{align}

Now, in the book, they assume that $\epsilon \sim \mathcal{N}(0, \sigma)$, so $\mathbb{E}\left[ \epsilon \right] = 0$ (i.e. the expected value of $\epsilon$ is just the mean of the Gaussian, which is assumed to be zero). So, \ref{5} becomes

\begin{align} &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( \mathbb{E} \left[ f(X) \right] + \mathbb{E} \left[ \epsilon \right] \right) + \hat{Y}^2 = \\ &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( f(X) + 0 \right) + \hat{Y}^2 = \\ &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ &\mathbb{E} \left[ f(X)^2 + 2 f(X) \epsilon + \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ &\mathbb{E} \left[ f(X)^2 \right] + \mathbb{E} \left[ 2 f(X) \epsilon \right] + \mathbb{E} \left[ \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ & \mathbb{E} \left[ \epsilon^2 \right] + f(X)^2 - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ & \mathbb{E} \left[ \epsilon^2 \right] + \left(f(X) - \hat{Y} \right)^2 \tag{6}\label{6} \end{align}

Now, note that the variance of a random variable $Z$ is defined as

$$\operatorname {Var} (Z)=\mathbb {E} \left[(Z - \mu_Z )^{2}\right]$$

In our case, $\mu_Z$ is zero, so the variance of $\epsilon$ is $\mathbb{E} \left[ \epsilon^2 \right]$, so \ref{6} becomes

\begin{align} \operatorname{Var} (\epsilon) + \left(f(X) - \hat{Y} \right)^2 \\ \tag{7}\label{7} \end{align}

You can also come up with the same result in a different and simpler way, i.e. rewrite $\mathbb{E}\left[ \left( f(X) + \epsilon - \hat f(X) \right)^2 \right]$ as $\mathbb{E}\left[ \left( \left(f(X) - \hat f(X)\right) +\epsilon \right)^2 \right]$, then you apply the distributive property and similar rules that I applied above to derive the same result.