When watching the machine learning course on Coursera by Andrew Ng, in the logistic regression week, the cost function was a bit more complex than the one for linear regression, but definitely not that hard.

But it got me thinking, why not use the same cost function for logistic regression?

So, the cost function would be $\frac{1}{2m} \sum_{i}^m|h(x_i) - y_i|^2$, where $h(x_i)$ is our hypothesis $\text{function}(\text{sigmoid}(X * \theta))$, $m$ is the number of training examples and $x_i$ and $y_i$ are our $ith$ training example?


The mean squared error (MSE), $J(\theta) = \frac{1}{2m}\sum_{i=1}^m(h_\theta(x_i)-y_i)^2$, is not as appropriate as a cost function for classification, given that the MSE makes assumptions about the data that are not appropriate for classification. Though, as an optimization objective, it is still possible to attempt to minimize MSE even in a classification problem, and thus still learn parameters $\theta$.

The new cost function has better convergence characteristics as it is more inline with the objective.

See link for the precise mathematical formulation that explains these loss functions from a probabilistic perspective.

Note that the absolute value is redundant because $\forall x:x^2\geq0$.

I hope this clarifies the matter.

  • 1
    $\begingroup$ MLE is also not guaranteed to be convex though. It's I guess 'relatively' more convex. $\endgroup$ – DuttaA Sep 5 at 3:39
  • 1
    $\begingroup$ @DuttaA OOF! you're right. Thanks for this comment! I did some reading on this and updated with a link that seems to provide the rigorous reasons. $\endgroup$ – respectful Sep 5 at 5:13

I mean you technically could (it's not going to break or something) however, cross entropy is much better suited for classification as it penalizes for misclassification errors: have a look at the function: when you are wrong the loss goes to infinity: enter image description here

you are either from one class or another. MSE is designed for regression where you have nuance: you get close to target is sometimes good enough. You should try both and you will see the performance will be much better for the cross entropy.

  • $\begingroup$ MSE also penalizes misclassification error, you could say it doesn't penalize it as much as MLE error function which might lead to the NN focusing on minimizing loss rather than increasing accuracy. $\endgroup$ – DuttaA Sep 5 at 11:30
  • $\begingroup$ I would not call it penalising misclassification per se, as it would be predicting probabilities but implicitely yes you're right I should have mentionned that $\endgroup$ – RonsenbergVI Sep 5 at 11:45
  • $\begingroup$ It doesn't predict probablities though. The target is either 1 or 0. Since its output is between 0-1 or we design it to be so (like in softmax) and also because I think conventional literature somewhat treats it in the same way, so we interpret it as a probablity. Case in point is SVMs which try to just escape from a region, but if we normalize it tgen it's just probability. While in SVM we will try to reach infinity hear reaching the same will result in a 1 and thus 0. But then again I think the viewpoint varies from person to person.(NOTE: you can always edit your answer to add more details) $\endgroup$ – DuttaA Sep 5 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.