How to understand marginal loglikelihood objective function as loss function (explanation of an article)?

Question

I am reading article https://allenai.org/paper-appendix/emnlp2017-wt/ http://ai2-website.s3.amazonaws.com/publications/wikitables.pdf about training neural network and the loss function is mentioned on page 6 chapter 3.4 - this loss function O(theta) is expressed as marginal loglikelihood objective function. I simply does not understand this. The neural network generates logical expression (query) from some question in natural language. The network is trained using question-answer pairs. One could expect that simple sum of correct-1/incorrect=0 result could be good loss function. But there is strange expression that involves P(l|qi, Ti; theta) that is not mentioned in the article. What is meant by this P function? As I understand, then many logical forms l are generated externally for some question qi. But further I can not understand this. The mentioned article largely builds on other article http://www.aclweb.org/anthology/P16-1003 from which it borrows some terms and ideas.

It is said that l is treated as latent variable and P seems to be some kind of probability. Of course, we should assign the greated probability to the right logical form l, but where can I find this assignment. Does training/supervision data should contain this probability function for training/supervision data?

Answer 1

I cannot answer your question but I am stuck in a similar rabbit hole so hopefully these references can help you.

The loss function you are describing would be 0-1 loss. However, 0 would be the if our output matches and 1 would be if it does not. This function is not smooth and not convex. Thus we often replace it with a surrogate loss function such as log likelihood.

You can read more about surrogate loss function on pg 269 of Deep Learning by Ian Goodfellow available here: https://www.deeplearningbook.org/

or here : https://en.wikipedia.org/wiki/Loss_functions_for_classification

I did not have time to read the article, but the reason that you are seeing a probability is that they are using a Bayesian framework for neural networks. This was first described in A Practical Bayesian Framework for Backpropagation networks by McKay 1992.

It is explained very well in a video series by Hinton on youtube here: https://www.youtube.com/watch?v=YcwZFNd3UvI

Hopefully these references help you I wish I could explain in greater detail.