# Are neural networks statistical models?

By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models.

I am looking for a more concise/summarized answer with focus on ANNs.

What is a statistical model?

According to Anthony C. Davison (in the book Statistical Models), a statistical model is a probability distribution constructed to enable inferences to be drawn or decisions made from data. The probability distribution represents the variability of the data.

Are all neural networks statistical models?

All neural networks that construct a probability distribution to draw inferences from the data or to make decisions from the data are statistical models.

Variational auto-encoders (VAEs) construct a probability distribution (e.g. a Gaussian) to draw inferences, so VAEs can be considered statistical models.

On the other hand, for example, MLPs do not necessarily construct any probability distribution, so they are not necessarily statistical models. However, note that MLPs can be used to represent the parameters of a distribution. For example, you could train a MLP to represent the mean of a Gaussian distribution. See e.g. Junction Tree Variational Autoencoder for Molecular Graph Generation for an example.

Consequently, not all neural networks are statistical models (at least, according to the definition by Davison).

• The few research papers I have read have all treated the NN (classifier) as producing a probablity distribution. Although I don't know whether they treat it as such to suit their job at hand. – DuttaA May 16 at 15:08
• @DuttaA Do you mean that they use like a softmax output layer? – nbro May 16 at 15:19
• I don't think so, but I might be wrong since I was not very clear as I had the doubt 'what kind of probablity distribution (bernoulii, etc) does an NN actually give if we consider the output to be a probablity distribution'. – DuttaA May 16 at 16:49
• @DuttaA Can you please link us to one of those papers? I am curious now. I'd like to know the terminology used there. – nbro May 16 at 16:50
• I think Unsupervised Learning using VAE by kingma is one. They defined the output as a multinomial probablity distribution (the doubt is why multinomial and not anything else). – DuttaA May 16 at 16:57

The question is a good one, and a necessary one to analyze for the good of AI, this site, and everyone doing research.

After reading the two articles and having previously read many others on how artificial networks are just curve fitting or that they are the way to general intelligence, there is very little educational value to gain. The arguments are specious, and the truth is easily within what a 3.0 math GPA high school student can grasp. Disambiguating the nonsense will require several paragraphs, so it is unrealistic to expect both concise and rational in the same answer.

Dependencies in the Development

Statistics was spawned from earlier mathematics. We know this because collecting data without any concept of probability is like a child collecting leafs. She or he may enjoy doing it and find the leafs beautiful, but the leaf collection will not have any broad use.

Probability theory necessarily preceded statistics, and arithmetic made probability theory practical. Francis Galton, Karl Pearson, and others made statistics practical by employing the algebraic representations of probability to acquired sets of data, so we have this:

$$\text{Arithmetic} \; \Rightarrow \; \text{Algebraic expression of probability} \, (P) \\ \Downarrow \\ P \; \cup \, \text{data acquisition} \; \Rightarrow \; \text{statistics.}$$

What's a statistical model but a model that employs statistics. Therefore it necessarily dependent upon probability and algebra, whether explicitly in the literature or through conventions that may obscure earlier proofs. Since Hilbert and others, we now have calculus and probability together, which in the AI literature is related to expectation and involves probability distributions. If those distributions involve curved lines or surfaces, calculus is required.

Modelling

What is a model other than what John von Neumann so famously and unapologetically stated.

The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical construct which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work.

— Method in the Physical Sciences, in The Unity of Knowledge, 1955, Doubleday, p. 157

Realistic and Concise Summary

Are neural networks statistical models?

Not exclusively. Artificial networks are models that employ mathematics, a portion of which is statistical because data sets are used to drive the convergence of the parameters of the artificial network to an optimum. The optimum is expressed by loss, error, value, or gain functions that aggregate to a single number the disparity between any instantaneous result during training and some ideal. It is essentially a Hilbert space transform that aggregates the criteria for acceptable convergence and the curvature of paths to it.

Since the transform and the ideal itself are curved, otherwise the use of an artificial network would be gross overkill, calculus is necessarily involved.

If the learning is continuous or re-entrant, meaning that it happens in parallel with the use of what has so far been learned or in time sliced series with it, then probability and calculus are further employed to deal with these kinds of mid-learning dynamics.

• Dynamic ideal that the converging system is expected to track
• Dynamic degree of information available about the disparity between the current state and the ideal (partial observation)

List of Contributing Disciplines

The theory behind artificial networks and the parallel mechanisms that may be realized in VLSI circuitry or in software algorithms depend on many areas of discipline.

• Logical inference
• Boolean algebra
• Algebra in $$\mathbb{R}^n$$
• Analytic geometry
• Finite series
• Newton's method and its refinements in convergence theory
• Calculus
• Probability
• Statistics
• Hardware architecture
• Algorithm design
• Graph representations
• Markov process
• Linguistics
• Control theory
• Cognitive concepts like attention
• Digital design
• Theory of central processing units
• Programming languages
• IEEE floating point standards
• Cantor sets
• Poincaré and Mendelbrot
• Shannon information theory and entropy

This list grows as other types of artificial networks are developed1 using interdisciplinary approaches.

Statistics cannot lay claim to a unique influence on artificial network design and use. No one of the various schools of thinking or areas of mathematical expertise can legitimately usurp the claims of the other contributing groups.

A Balanced Perspective

One point that brings machine learning back to its humble center is that the simplest form of network training is clearly curved surface fitting using an orthogonal, parametric model. This is the MLP and has these characteristics.

• An arbitrary model intended to approximate complexity
• Convergence on this arbitrary curved function driven by a data set
• Quadratic summing used to aggregate disparity as a scalar

This is legitimately a statistical process. However, it is so among other things.

Is machine learning2 glorified statistics?

Using that same dismissal pattern, we could then say a number of other things.

• Statistics is glorified probability theory.
• Calculus is glorified algebra.
• Supercomputers are glorified calculators.
• Google is a glorified librarian and wooden card catalog.

But none of those dismissal patterns honor the contributions of research, which is often misquoted, misunderstood, not at all glorified, marginally appreciated, and frequently underpaid.

Machine learning doesn't include Fuzzy logic controllers. This odd controller type isn't use anymore in academia at all, especially not for statistical models.

The editors of the Journal of Machine Learning disagree. They published at least ten articles that expound upon the topic "fuzzy logic" and the idea of fuzzy rules appears in dozens more of their publications. Furthermore, a fuzzy logic controller is definitely a Turing machine and it learns via some of the very same mechanisms that artificial networks use to learn. If FL isn't ML, then DL isn't either. Lastly, the AI goal of cognition may be best implemented using fuzzy inference.

Footnotes

[1] Many types of artificial networks have emerged, and each of these few examples listed here have variants.

• MLP — multi-layer perceptrons to use the historical reference
• FFN — feed forward network
• LSTM — long short term memory (a network cell type)
• GRU — gated recurrent unit (a network cell type)
• RBF — radial basis function
• KSO — Kohonen self organizing
• ConvNet — one of the types of convolution kernel based networks
• RDN — recursive deep networks used for semantic structures

[2] ML ≉ ANNs in the AI literature. Equating the two is becoming less common in online content too, and understandably so.

One reason is that the neurological understanding that influenced the MLP design is obsolete in at least a dozen ways, including neuroplasticity, brain chemistry, and axon energy supply phenomena. Even excluding these, the perceptron was a convenient algebraic model and did intend to model the pulse dynamics of real neurons where signals ran in parallel and could meet at a single neuron with the complexity of phase misalignment. These pioneering artificial nets are not neural, although attention devices, neuromorphic artificial networks, and gating have slightly improved that shortcoming.

ML also goes beyond networks and includes fuzzy logic controllers, self-calibrating industrial PID controllers, production systems with meta rules that acquire rules through genetic algorithms, and many other technologies.

• Machine learning doesn't include Fuzzy logic controllers. This odd controller type isn't use anymore in academia at all, especially not for statistical models. – Manuel Rodriguez Jan 29 at 17:13
• Clear detailed answer. (+1) – naive May 16 at 14:13
• I think it does answer the question. -- which is Are neural networks statistical models? To which the answer posted is Not exclusively which in my opinion is true. 'Detailed' because it provides backing information for the answer. – naive May 16 at 14:29
• @DouglasDaseeco, You again took revenge on me because I down-voted some of your answers, which I think are not good enough. I will flag you again for voting irregularities and you will be banned again (but for a longer period of time). Enjoy. – nbro May 16 at 15:03
• @DouglasDaseeco Ahead of you? I study AI, so I more or less know what I am talking about on this website. I downvoted the answers that I think are not good enough for this website. Your answers are rarely on point and are often unnecessarily long. This is not a website to write blog posts. Why don't you start a blog? I think that format is more appropriate for your way of writing. – nbro May 16 at 16:10

Neural network can be best compared with logic gates. A logic gate is network of boolean connectors which are and/or/not. After putting an input signal to the network, the output signal is calculated. In contrast to a plain statistical model, logic gates and also neural networks are turing powerful, which means, they can produce more complex patterns which involves repetitions and programs.

The best example which shows the similarity between neural networks and logic gates is a calculator. The input signals are “1”, “2”, “+” and the network calculates the correct result which is “3”. Logic gates and advanced neural network can do both the task. The difference is, that logic gates are created manually which is called cpu-design, while neural networks are trained with example data.

A logic gate is not that the same like a fuzzy logic gate. A logic gate can only contains zeros and one values but not numbers in between. Logic gates and neural networks have only the ability to express boolean logic which is based on true or false. That means, a single neuron can take a binary decision to fire or not.