2
$\begingroup$

I am studying logistic regression for binary classification.

The loss function used is cross-entropy. For a given input $x$, if our model outputs $\hat{y}$ instead of $y$, the loss is given by $$\text{L}_{\text{CE}}(y,\hat{y}) = -[y \log \hat{y} + (1 - y) (\log{1 - \hat{y}})]$$

Suppose there are $m$ such training examples, then the overall total loss function $\text{TL}_{\text{CE}}$ is given by

$$\text{TL}_{\text{CE}} = \dfrac{1}{m} \sum\limits_{i = 1}^{m} \text{L}_{\text{CE}} (y_i , \hat{y_i}) $$

It is said that the loss function is convex. That is, If I draw a graph between the loss values wrt the corresponding weights then the curve will be convex. The material from textbook did not give any explanation regarding the convex nature of the cross entropy loss function. You can observe it from the following passage.

For logistic regression, this (cross-entropy) loss function is conveniently convex. A convex function has just one minimum; there are no local minima to get stuck in, so gradient descent starting from any point is guaranteed to find the minimum. (By contrast,the loss for multi-layer neural networks is non-convex, and gradient descent may get stuck in local minima for neural network training and never find the global optimum.)

How did they conclude conveniently that the loss function is convex? Is it by plotting or some other means?

$\endgroup$
1
$\begingroup$

If you find the Hessian matrix (the matrix of second order derivatives) for the binary cross entropy loss function, you'll see that it is positive semidefinite for any possible value of the parameters. This concludes that it is a convex function.

A side effect of it being convex is that it will have a single minimum as mentioned in the textbook you cited.

$\endgroup$
1
  • 1
    $\begingroup$ Maybe you could at least provide a link to a reference that shows this more in detail. For people also not very familiar with linear algebra, you could also explain a little bit more the relationship between the convexivity and being or not positive semi-definite (and what this latter even means). $\endgroup$
    – nbro
    Jun 21 at 22:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.