Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
E. Alpaydin, Introduction to ML, 4-th edition, page 46:
Over time, it has been realized in the neural network community that most neural network learning algorithms have their basis in statistics—for example, the multilayer perceptron is another class of nonparametric estimator.
Why is multilayer perceptron a nonparametric model?
“Nonparametric” is a controversial term in statistics whose definition lacks a consensus.
One definition is that the parameter count increases as the data size increases. This is how kernel density estimation works: for each new point, we add a new kernel. Perhaps a neural network could be seen in this light by allowing the number of neurons to increase as data are added.
One definition is that we allow the data to describe themselves rather than shoehorning them into a particular form that is governed by parameters. A neural network could be seen this way, as the idea is to give a large amount of flexibility to allow the data to extract their own features that involve curvature and variable interactions; we don’t have to specify these like we do in a linear regression or generalized linear model if the model figures them out.
If the author had referred to a neural network as a particular type of nonlinear regression, then the statement would not be controversial.
Because of the lack of consensus on what “nonparametric” means, towards the end of 2020, there was a movement/joke on Cross Validated (Statistics Stack Exchange) that our new year’s resolution would be to stop using the term “nonparametric” for exactly that reason.
