By reading the abstract of Neural Networks and Statistical Models paper it would seem that ANNs are statistical models.
In contrast Machine Learning is not just glorified Statistics.
I am looking for a more concise/summarized answer with focus on ANNs.
What is a statistical model?
According to Anthony C. Davison (in the book Statistical Models), a statistical model is a probability distribution constructed to enable inferences to be drawn or decisions made from data. The probability distribution represents the variability of the data.
Are all neural networks statistical models?
All neural networks that construct a probability distribution to draw inferences from the data or to make decisions from the data are statistical models.
Variational auto-encoders (VAEs) construct a probability distribution (e.g. a Gaussian) to draw inferences, so VAEs can be considered statistical models.
There are also Bayesian neural networks, which are neural networks that maintain a probability distribution (usually, a Gaussian) for each unit (or neuron) of the neural network, rather than only a point estimate. BNNs can thus also be considered statistical models.
On the other hand, for example, MLPs do not necessarily construct any probability distribution, so they are not necessarily statistical models. However, note that MLPs can be used to represent the parameters of a distribution. For example, you could train an MLP to represent the mean of a Gaussian distribution. See e.g. Junction Tree Variational Autoencoder for Molecular Graph Generation for an example.
Consequently, not all neural networks are statistical models (at least, according to the definition by Davison).
According to Wikipedia:
A statistical model is a mathematical model that embodies a set of statistical assumptions concerning the generation of sample data (and similar data from a larger population). A statistical model represents, often in considerably idealized form, the data-generating process.
Answer to your question:
To build any neural network model we assume the train, test and validation data are coming from a probability distribution. So, if you produce a neural network model based on statistical data then the network is a statistical model.
Moreover, neural networks' cost function is generally a parametric model and parametric modes are statistical models.
Please look at Goodfellow's Deep Learning book chapter Deep Feedforward Networks page 174 and 175.
From Goodfellow's book
Fortunately, the cost functions for neural networks are more or less the same as those for other parametric models, such as linear models. In most cases, our parametric model defines a distribution $p(y \mid x; \theta)$ and we simply use the principle of maximum likelihood.
In conclusion, ANNs (e. g. MLP, CNN, etc.) are statistical models
