Modern artificial neural networks use a lot more functions than just the classic sigmoid, to the point I'm having a hard time really seeing what classifies something as a "neural network" over other function approximators (such as Fourier series, Bernstein polynomials, Chebyshev polynomials or splines).

So, what makes something an artificial neural network? Is there a subset of theorems that apply only to neural networks?

Backpropagation is classic, but that is the multi-variable chain rule, what else is unique to neural networks over other function approximators?


First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule).

The first difference between neural networks and other function approximators is conceptual. In neural networks, you typically imagine that there are one or more computational units (often called neurons) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of flexibility and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks.

The other difference is that neural networks were originally inspired by the biological counterparts. See A logical calculus of the ideas immanent in nervous activity (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.

There are other technical differences. For example, the Taylor expansion of a function is typically done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the Fourier series can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (at least, this is the setup in the famous paper that proved the universality of neural networks) in many different ways (for example, weighted sums followed by sigmoids).

To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.