Can the mean squared error be negative?

Question

I'm new to machine learning. I was watching a Prof. Andrew Ng's video about gradient descent from the machine learning online course. It said that we want our cost function (in this case, the mean squared error) to have the minimum value, but that minimum value shown in the graph was not 0. It was a negative number!

How can our cost function, which is mean squared error, have a negative value, given that the square of a real number is always positive? Even if it is possible, don't we want our error to be 0?

Answer 1

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.

A standard Mean Squared Error function cannot be negative. The lowest possible value is $0$, when there is no output error from any example input.

How can our cost function which is mean squared error have a value under 0?

It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for any cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.

Many loss or cost functions are designed with an absolute minimum of $0$ possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.

Answer 2

Mean squared error terms must be positive because the square of a number is positive. Therefore the sum (cost) is positive. The error is the difference between the hypothesis and the observation.

I would focus on understanding why Ng seeks to minimize J and how it is minimization is achieved with partial derivatives via matrix implementation.

Answer 3

I am watching the same course too, and I think that in the example graph, the cost function is not a sum of MSE (Mean squarred errors), but it could be a cubic one, so a sum of cubical errors, and thus the cost function could be negative: as there are a variety of cost functions, the MSE ones are not adapted for every problems, and other formulations could work better.