Can the mean squared error be negative?

Question

I'm new to machine learning. I was watching a Prof. Andrew Ng's video about gradient descent from the machine learning online course. It said that we want our cost function (in this case, the mean squared error) to have the minimum value, but that minimum value shown in the graph was not 0. It was a negative number!

How can our cost function, which is mean squared error, have a negative value, given that the square of a real number is always positive? Even if it is possible, don't we want our error to be 0?

Answer 1

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.

A standard Mean Squared Error function cannot be negative. The lowest possible value is $0$, when there is no output error from any example input.

How can our cost function which is mean squared error have a value under 0?

It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for any cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.

Many loss or cost functions are designed with an absolute minimum of $0$ possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.

Answer 2

Mean squared error terms must be positive because the square of a number is positive. Therefore the sum (cost) is positive. The error is the difference between the hypothesis and the observation.

I would focus on understanding why Ng seeks to minimize J and how it is minimization is achieved with partial derivatives via matrix implementation.

Answer 3

I am watching the same course too, and I think that in the example graph, the cost function is not a sum of MSE (Mean squarred errors), but it could be a cubic one, so a sum of cubical errors, and thus the cost function could be negative: as there are a variety of cost functions, the MSE ones are not adapted for every problems, and other formulations could work better.

Answer 4

I would like to add more information to the @Neil Slater's answer.

Andrew sir has used a "generic" graph to show min loss function mean higher performance. That is not always the case, sometimes its over fitting. Let's no go into that realm of machine learning I am not that pro

The output of the function in maths is called range and input is called domain. We are using MSE in the linear regression because it is sensitive to the outliers and help in penalizing the parameters to get more accurate value that fits the training example (aka line of best fit).

In the MSE, the minimum loss would be $\hat y$ = y which means the difference is $0$, so does the mean $\mu$ would be zero. Therefore the range of function would be $[0, \infty)$. In case of MSE, the min value of the cost function would be $0$ no matter what.

Do not confuse the $0$ min value of the MSE with the min $0$ of gradient descent of the cost function. Remember, in the gradient descent there is no 2 in the power, it can be negative, despite being positive value of the MSE function.