3
$\begingroup$

When watching the machine learning course on Coursera by Andrew Ng, in the logistic regression week, the cost function was a bit more complex than the one for linear regression, but definitely not that hard.

But it got me thinking, why not use the same cost function for logistic regression?

So, the cost function would be $\frac{1}{2m} \sum_{i}^m|h(x_i) - y_i|^2$, where $h(x_i)$ is our hypothesis $\text{function}(\text{sigmoid}(X * \theta))$, $m$ is the number of training examples and $x_i$ and $y_i$ are our $ith$ training example?

$\endgroup$
4
$\begingroup$

The mean squared error (MSE), $J(\theta) = \frac{1}{2m}\sum_{i=1}^m(h_\theta(x_i)-y_i)^2$, is not as appropriate as a cost function for classification, given that the MSE makes assumptions about the data that are not appropriate for classification. Though, as an optimization objective, it is still possible to attempt to minimize MSE even in a classification problem, and thus still learn parameters $\theta$.

The new cost function has better convergence characteristics as it is more inline with the objective.

See link for the precise mathematical formulation that explains these loss functions from a probabilistic perspective.

Note that the absolute value is redundant because $\forall x:x^2\geq0$.

I hope this clarifies the matter.

$\endgroup$
2
  • 1
    $\begingroup$ MLE is also not guaranteed to be convex though. It's I guess 'relatively' more convex. $\endgroup$
    – user9947
    Sep 5 '19 at 3:39
  • 1
    $\begingroup$ @DuttaA OOF! you're right. Thanks for this comment! I did some reading on this and updated with a link that seems to provide the rigorous reasons. $\endgroup$
    – respectful
    Sep 5 '19 at 5:13
1
$\begingroup$

I mean you technically could (it's not going to break or something) however, cross entropy is much better suited for classification as it penalizes for misclassification errors: have a look at the function: when you are wrong the loss goes to infinity: enter image description here

you are either from one class or another. MSE is designed for regression where you have nuance: you get close to target is sometimes good enough. You should try both and you will see the performance will be much better for the cross entropy.

$\endgroup$
3
  • $\begingroup$ MSE also penalizes misclassification error, you could say it doesn't penalize it as much as MLE error function which might lead to the NN focusing on minimizing loss rather than increasing accuracy. $\endgroup$
    – user9947
    Sep 5 '19 at 11:30
  • $\begingroup$ I would not call it penalising misclassification per se, as it would be predicting probabilities but implicitely yes you're right I should have mentionned that $\endgroup$ Sep 5 '19 at 11:45
  • $\begingroup$ It doesn't predict probablities though. The target is either 1 or 0. Since its output is between 0-1 or we design it to be so (like in softmax) and also because I think conventional literature somewhat treats it in the same way, so we interpret it as a probablity. Case in point is SVMs which try to just escape from a region, but if we normalize it tgen it's just probability. While in SVM we will try to reach infinity hear reaching the same will result in a 1 and thus 0. But then again I think the viewpoint varies from person to person.(NOTE: you can always edit your answer to add more details) $\endgroup$
    – user9947
    Sep 5 '19 at 14:08
0
$\begingroup$

Minimising MSE in a classification setting is perfectly reasonable as it is also known as the Brier Score and is a proper scoring rule which means that it is minimised if the network outputs the conditional probability of class membership. This is not unduly surprising as minimising MSE leads to a model that outputs an estimate of the conditional mean of the target distribution, which for a 1-of-c coding is the conditional probability of class membership. You can even use MSE for training networks with logistic or softmax activation functions in the output layer so that the obey the usual constraints of being in the interval $[0,1]$ and summing to one.

However, the MSE penalises very confident misclassifications much less harshly than the cross-entropy metric does. Whether this is a good or bad thing depends on the needs of the application. If you are mostly interested in the p=0.5 decision boundary, then you probably don't want model resources spent dealing with highly confident misclassifications, which are a long way from the decision boundary, and have little effect on it. This is a large part of the justification for purely discriminative methods like the SVM.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy