In the back-propogation algorithm, the error term is:

$$ E=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2, $$

where $\hat{y}_k$ is a vector of outputs from the network, $y_k$ is the vector of correct labels (and we work out the error by calculating predicted-observed, squaring the answer, and then summing the answers for each $k$ (and dividing by 2).

How do you prove that if this answer is $0$ (i.e., if $E=0$), then $\hat{y}_k=y_k$ for all $k$?


This is very easy to prove.

Let's first prove that, if $\hat{y}_k = y_k$, then the $E = 0$. I will leave all steps, so that it's super clear.

\begin{align} E &=\frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 \\ &=\frac{1}{2}\sum_k(y_k - y_k)^2\\ &=\frac{1}{2}\sum_k(0)^2\\ &=\frac{1}{2}\sum_k 0\\ &=\frac{1}{2} 0\\ &=0\\ \end{align}

To prove the other way around, i.e. if $E = 0$, then $\hat{y}_k = y_k$, you can do as follows

\begin{align} \frac{1}{2}\sum_k(\hat{y}_k - y_k)^2 &=E\\ &=0 \end{align} Recall now that any number squared is non-negative (i.e. positive or zero). Given that $(\hat{y}_k - y_k)^2 $ is non-negative, then $\sum_k(\hat{y}_k - y_k)^2$ is a sum of non-negative numbers. The only way that a sum of non-negative numbers is equal to zero is if all numbers are zero, so we must have $\hat{y}_k = y_k$ (because any non-zero number squared is non-zero).

(Note that $E$ is the mean squared error, i.e. a loss function, and it's not the back-propagation algorithm, which is just the algorithm that you use to compute partial derivatives of $E$ with respect to the parameters of the model, which are not even visible in the way you wrote $E$).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.