2
$\begingroup$

In the book An Introduction to Statistical Learning, the authors claim (equation 2.3, p. 19, chapter 2)

$$\mathbb{E} \left[ (Y - \hat{Y})^2 \right] = \left(f(X) - \hat{f}(X) \right)^2 + \operatorname{Var} (\epsilon) \label{0}\tag{0},$$

where

  • $Y = f(X) + \epsilon$, where $\epsilon \sim \mathcal{N}(0, \sigma)$ and $f$ is the unknown function we want to estimate
  • $\hat{Y} = \hat{f}(X)$ is the output of our estimate of $f$, i.e. $\hat{f} \approx f$

They claim that this is easy to prove, but this may not be easy to prove for everyone. So, why is equation \ref{0} true?

$\endgroup$
2
  • 2
    $\begingroup$ This is missing a crucial assumption in the book that $X$ (and both $f$s) are fixed. Otherwise the equation is not true. $\endgroup$
    – Kostya
    May 16 at 19:11
  • $\begingroup$ @Kostya I'm saying that in my answer. $\endgroup$
    – nbro
    May 16 at 19:51
2
$\begingroup$

Let's say we have $a$ - constant and $\epsilon \sim \mathcal{N}(0,\sigma)$, then: $$\mathbb{E}\left[(a+\epsilon)^2\right] = \mathbb{E}\left[a^2\right] + 2 \mathbb{E}\left[a\right]\mathbb{E}\left[\epsilon\right] + \mathbb{E}\left[\epsilon^2\right] $$ Expectations of constants are just the constants: $\mathbb{E}[a] = a$ and $\mathbb{E}[a^2] = a^2$
The mean of $\epsilon$ is zero $\mathbb{E}[\epsilon] = 0$. And the expectation of $\epsilon^2$ is its variance: $$ \mathop{\mathrm{Var}}(\epsilon) = \mathbb{E}[\epsilon^2] - \mathbb{E}[\epsilon]^2 = \mathbb{E}[\epsilon^2]$$

Substituting, we get an expression for the original expectation:
$$\mathbb{E}\left[(a+\epsilon)^2\right] = a^2 + \mathop{\mathrm{Var}}(\epsilon) \tag{*}$$

Getting to the expectation in the book, we first substitute the values for $Y$ and $\hat{Y}$: $$\mathbb{E}\left[(Y - \hat{Y})^2\right] = \mathbb{E}\left[((f(X) - \hat{f}(X)) + \epsilon)^2\right]$$

In the book it is assumed that $X$, $f$ and $\hat{f}$ are constant. So we can use the expression (*) with the constant being $a = f(X) - \hat{f}(X)$:

$$\mathbb{E}\left[(Y - \hat{Y})^2\right] = (f(X) - \hat{f}(X))^2 + \mathop{\mathrm{Var}}(\epsilon)$$

$\endgroup$
0
1
$\begingroup$

Let me try to show this. The only (non-constant) random variable here is $\epsilon$, while $f(X)$ and $\hat{Y} = \hat{f}(X)$ are constant random variables (so their expectations is equal to their only value).

So, we start with the following expression.

\begin{align} \mathbb{E} \left[ (Y - \hat{Y})^2 \right] \tag{1}\label{1} \end{align}

Now, we just apply the distributive property, so \ref{1} becomes

\begin{align} \mathbb{E} \left[ Y^2 - 2Y \hat{Y} + \hat{Y}^2 \right] \tag{2}\label{2} \end{align}

Given the linearity of the expectation, we can write \ref{2} as follows

\begin{align} \mathbb{E} \left[ Y^2 \right] - \mathbb{E} \left[ 2Y \hat{Y} \right] + \mathbb{E} \left[\hat{Y}^2 \right] \tag{3}\label{3} \end{align}

Given that $\hat{Y} = \hat{f}(X)$ is a constant and that we can take constants out of the expectations, we have

\begin{align} \mathbb{E} \left[ Y^2 \right] - 2 \hat{Y} \mathbb{E} \left[ Y \right] + \hat{Y}^2 \tag{4}\label{4} \end{align}

Now, let's replace $Y$ with $f(X) + \epsilon$, to obtain

\begin{align} \mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \mathbb{E} \left[ f(X) + \epsilon \right] + \hat{Y}^2 \tag{5}\label{5} \end{align}

Now, in the book, they assume that $\epsilon \sim \mathcal{N}(0, \sigma)$, so $\mathbb{E}\left[ \epsilon \right] = 0$ (i.e. the expected value of $\epsilon$ is just the mean of the Gaussian, which is assumed to be zero). So, \ref{5} becomes

\begin{align} &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( \mathbb{E} \left[ f(X) \right] + \mathbb{E} \left[ \epsilon \right] \right) + \hat{Y}^2 = \\ &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} \left( f(X) + 0 \right) + \hat{Y}^2 = \\ &\mathbb{E} \left[ \left( f(X) + \epsilon \right)^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ &\mathbb{E} \left[ f(X)^2 + 2 f(X) \epsilon + \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ &\mathbb{E} \left[ f(X)^2 \right] + \mathbb{E} \left[ 2 f(X) \epsilon \right] + \mathbb{E} \left[ \epsilon^2 \right] - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ & \mathbb{E} \left[ \epsilon^2 \right] + f(X)^2 - 2 \hat{Y} f(X) + \hat{Y}^2 = \\ & \mathbb{E} \left[ \epsilon^2 \right] + \left(f(X) - \hat{Y} \right)^2 \tag{6}\label{6} \end{align}

Now, note that the variance of a random variable $Z$ is defined as

$$\operatorname {Var} (Z)=\mathbb {E} \left[(Z - \mu_Z )^{2}\right]$$

In our case, $\mu_Z$ is zero, so the variance of $\epsilon$ is $\mathbb{E} \left[ \epsilon^2 \right]$, so \ref{6} becomes

\begin{align} \operatorname{Var} (\epsilon) + \left(f(X) - \hat{Y} \right)^2 \\ \tag{7}\label{7} \end{align}

You can also come up with the same result in a different and simpler way, i.e. rewrite $\mathbb{E}\left[ \left( f(X) + \epsilon - \hat f(X) \right)^2 \right]$ as $\mathbb{E}\left[ \left( \left(f(X) - \hat f(X)\right) +\epsilon \right)^2 \right]$, then you apply the distributive property and similar rules that I applied above to derive the same result.

$\endgroup$
1
  • $\begingroup$ Meanwhile, I also realized that this question was also asked here. You can also find the proofs of equation 2.7 of the same book here. There's also this Wikipedia article that talks about the topic. $\endgroup$
    – nbro
    May 18 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.