Normally, in practice, people use those loss functions with minima, e.g. $L_1$ mean absolute loss, $L_2$ mean squared error, etc. All those come with a minimum to optimize to. enter image description here

However, there's another thing, logistic loss, I'm reading about, but don't get it why the logistic function could be used as a loss function, given that it has the so-called minimum at infinity, but that isn't a normal minimum. Logistic loss function (black curve):

enter image description here

How can an optimizer minimize the logistic loss?


I see why you might be confused. First, the logistic-loss or log-loss is technically called cross-entropy loss. This function is very simple:

$CE = -[y \log(p) + (1 - y) \log(1 - p)]$

This tells basically if the predicted class $y$ was right $y=1$ then the loss is $CE=-\log(p)$, if the predicted class was not the right one then the loss is $CE=-\log(1-p)$.

If we look at the function as a pure math concept we see that:

$CE = f(x) = - \log(x)$

And as you point out, that function is minima-unbounded as its domain is $D(f(x)) = [0, +\infty]$. You can check that in here:

enter image description here

However the trick is that the inputs must be bounded, meaning, the inputs to the loss function must be in range $[0, 1]$. This bounding is achieved by applying a sigmoid activation function as the final "layer" of the network. Then if the inputs to the loss function are bounded the function has a clear minima.

Check how the function looks in reality from one of the most important papers in loss function in the AI world: Focal Loss (I really encourage you to read it as the first section explains in detail the cross-entropy loss). The blue curve is the one you are looking for.

enter image description here

Finally, you might want to review your log-loss/CE function since it should have an asymptote for $f(x=0) = \infty$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.