By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models.

In contrast Machine Learning is not just glorified Statistics.

I am looking for a more concise/summarized answer with focus on ANNs.


What is a statistical model?

According to Anthony C. Davison (in the book Statistical Models), a statistical model is a probability distribution constructed to enable inferences to be drawn or decisions made from data. The probability distribution represents the variability of the data.

Are all neural networks statistical models?

All neural networks that construct a probability distribution to draw inferences from the data or to make decisions from the data are statistical models.

Variational auto-encoders (VAEs) construct a probability distribution (e.g. a Gaussian) to draw inferences, so VAEs can be considered statistical models.

On the other hand, for example, MLPs do not necessarily construct any probability distribution, so they are not necessarily statistical models. However, note that MLPs can be used to represent the parameters of a distribution. For example, you could train a MLP to represent the mean of a Gaussian distribution. See e.g. Junction Tree Variational Autoencoder for Molecular Graph Generation for an example.

Consequently, not all neural networks are statistical models (at least, according to the definition by Davison).

  • $\begingroup$ The few research papers I have read have all treated the NN (classifier) as producing a probablity distribution. Although I don't know whether they treat it as such to suit their job at hand. $\endgroup$ – DuttaA May 16 '19 at 15:08
  • $\begingroup$ @DuttaA Do you mean that they use like a softmax output layer? $\endgroup$ – nbro May 16 '19 at 15:19
  • $\begingroup$ I don't think so, but I might be wrong since I was not very clear as I had the doubt 'what kind of probablity distribution (bernoulii, etc) does an NN actually give if we consider the output to be a probablity distribution'. $\endgroup$ – DuttaA May 16 '19 at 16:49
  • $\begingroup$ @DuttaA Can you please link us to one of those papers? I am curious now. I'd like to know the terminology used there. $\endgroup$ – nbro May 16 '19 at 16:50
  • $\begingroup$ I think Unsupervised Learning using VAE by kingma is one. They defined the output as a multinomial probablity distribution (the doubt is why multinomial and not anything else). $\endgroup$ – DuttaA May 16 '19 at 16:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.