3
$\begingroup$

When watching the machine learning course on Coursera by Andrew Ng, in the logistic regression week, the cost function was a bit more complex than the one for linear regression, but definitely not that hard.

But it got me thinking, why not use the same cost function for logistic regression?

So, the cost function would be $\frac{1}{2m} \sum_{i}^m|h(x_i) - y_i|^2$, where $h(x_i)$ is our hypothesis $\text{function}(\text{sigmoid}(X * \theta))$, $m$ is the number of training examples and $x_i$ and $y_i$ are our $ith$ training example?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
4
$\begingroup$

The mean squared error (MSE), $J(\theta) = \frac{1}{2m}\sum_{i=1}^m(h_\theta(x_i)-y_i)^2$, is not as appropriate as a cost function for classification, given that the MSE makes assumptions about the data that are not appropriate for classification. Though, as an optimization objective, it is still possible to attempt to minimize MSE even in a classification problem, and thus still learn parameters $\theta$.

The new cost function has better convergence characteristics as it is more inline with the objective.

See link for the precise mathematical formulation that explains these loss functions from a probabilistic perspective.

Note that the absolute value is redundant because $\forall x:x^2\geq0$.

I hope this clarifies the matter.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ MLE is also not guaranteed to be convex though. It's I guess 'relatively' more convex. $\endgroup$
    – user9947
    Commented Sep 5, 2019 at 3:39
  • 1
    $\begingroup$ You're right. Thanks for this comment! I did some reading on this and updated with a link that seems to provide the rigorous reasons. $\endgroup$
    – respectful
    Commented Sep 5, 2019 at 5:13
  • $\begingroup$ The MSE is minimised by the conditional mean of the targets. If the targets are {0,1} then the conditional mean will asymptotically be the probability of the label being 1, which means it is a "proper scoring rule" and suitable for classification. It is used in a wide range of classifiers (e.g. Fishers Linear Discriminant Analysis, Least-Squares Support Vector Machine and the L2 SVM is equivalent to regularised linear regression on the class labels of the set of support vectors). $\endgroup$
    – Dikran Marsupial
    Commented Oct 30 at 12:31
1
$\begingroup$

I mean you technically could (it's not going to break or something) however, cross entropy is much better suited for classification as it penalizes for misclassification errors: have a look at the function: when you are wrong the loss goes to infinity: enter image description here

you are either from one class or another. MSE is designed for regression where you have nuance: you get close to target is sometimes good enough. You should try both and you will see the performance will be much better for the cross entropy.

Improve this answer
$\endgroup$
3
  • $\begingroup$ MSE also penalizes misclassification error, you could say it doesn't penalize it as much as MLE error function which might lead to the NN focusing on minimizing loss rather than increasing accuracy. $\endgroup$
    – user9947
    Commented Sep 5, 2019 at 11:30
  • $\begingroup$ I would not call it penalising misclassification per se, as it would be predicting probabilities but implicitely yes you're right I should have mentionned that $\endgroup$
    – RonsenbergVI
    Commented Sep 5, 2019 at 11:45
  • $\begingroup$ It doesn't predict probablities though. The target is either 1 or 0. Since its output is between 0-1 or we design it to be so (like in softmax) and also because I think conventional literature somewhat treats it in the same way, so we interpret it as a probablity. Case in point is SVMs which try to just escape from a region, but if we normalize it tgen it's just probability. While in SVM we will try to reach infinity hear reaching the same will result in a 1 and thus 0. But then again I think the viewpoint varies from person to person.(NOTE: you can always edit your answer to add more details) $\endgroup$
    – user9947
    Commented Sep 5, 2019 at 14:08
1
$\begingroup$

Minimising MSE in a classification setting is perfectly reasonable as it is also known as the Brier Score and is a proper scoring rule which means that it is minimised if the network outputs the conditional probability of class membership. This is not unduly surprising as minimising MSE leads to a model that outputs an estimate of the conditional mean of the target distribution, which for a 1-of-c coding is the conditional probability of class membership. You can even use MSE for training networks with logistic or softmax activation functions in the output layer so that the obey the usual constraints of being in the interval $[0,1]$ and summing to one.

However, the MSE penalises very confident misclassifications much less harshly than the cross-entropy metric does. Whether this is a good or bad thing depends on the needs of the application. If you are mostly interested in the p=0.5 decision boundary, then you probably don't want model resources spent dealing with highly confident misclassifications, which are a long way from the decision boundary, and have little effect on it. This is a large part of the justification for purely discriminative methods like the SVM.

Improve this answer
$\endgroup$
0
$\begingroup$

This is also asked here

My response there is specific to MSE with sigmoid evaluation of the input: (y - sigmoid(X))^2 simply to keep the range of input evaluation within 0 and 1.

Improve this answer
$\endgroup$

You must log in to answer this question.