I am trying to train a supervised model where the output from the model is output of a linear function $WX + b$. Kindly note that I'm not using any softmax or $\log$ softmax on the result of the linear. I am using negative log-likelihood loss function, which takes the input as the linear output from the model and the true labels. I am getting decent accuracy by doing this, but I have read that the input to negative log-likelihood function must be probabilities. Am I doing something wrong?

  • $\begingroup$ log softmax is used for classification problems only. Are you solving a classification problem? If not you should use a MSEloss or something for regression task. Apart from that softmax takes the network output and transforms it to probabilities. NLL just takes the negative logarithm. So for classification you usually combine a softmax followed by NLL. However, in praxis you often do this in one go that avoids explicit calculation of steps inbetween for better numerical stability, such combined function is often called CrossEntropy loss. So take that for a classifica. task with raw net output. $\endgroup$ Dec 30 '18 at 14:06

This seems pretty reasonable to me. You can optimize for any function that is proportionate to the negative log-likelihood. Conventionally, we assume that the likelihood of piece of data under a linear model is proportionate to some sort of gaussian function of the difference between the predicted value and the observed value. If you're a Bayesian you'd say this is a probability. If you're a hardcore frequentist, you might quibble about that, but it's still a number between 0 and 1.

If you take the log of this likelihood function however, you'll get a quadratic function of the difference, with some scalers that you don't need to worry about. So you ought to minimize:

$$\sum_y (y-\hat{y})^2$$

This is not a probability, but since it is proportionate to the log of the original likelihood function (which was, in some sense), minimizing it will also minimize the original function.

Hope that helps!

  • $\begingroup$ Thanks for the help. It made some sense to me but still, I am not able to understand completely. It would be nice if you could provide some sources/supplements that would help me in having a better understanding of your answer. $\endgroup$ Sep 2 '18 at 7:16
  • $\begingroup$ This is a reasonable starting place: en.wikipedia.org/wiki/… $\endgroup$ Sep 2 '18 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.