This article attempts to provide a graphical justification of the universal approximation theorem.

It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded constant function or step function, and thus can therefore to a reasonable degree of approximation produce any function by essentially splitting up any function into a cluster (linear combination?) of these towers or steps.

However, he produced the steps and towers using specific weight parametrizations.

However, since when are we allowed to specify weights and biases? Isn't this all out of our hands and in the hands of cost function minimization?

I don't understand why he was dealing with setting weights to this, biases to that, when in my experience that is all done by "the machine" to minimize the cost function. I doubt the weights to minimize the cost function are arranged in the ways specified in order to form the towers and steps that were formed in this tutorial, so I kind of don't understand what all the hub-ub is all about.


The classical version of the universal approximation theorem states that, roughly, given a continuous function $f \colon [0, 1]^n \to [0, 1]^n$, there exists a single layer neural network and a set of weights and biases such that this network approximates the given function $f$ arbitrarily well.

It doesn't say anything about how you obtain such weights: the result is entirely independent of the way you train your network, and what it says is that the set of single-layer neural networks has enough capacity, in principle, to approximate any continuous function arbitrarily well.

This kind of result is fairly common in mathematics: it is shown that a suitable object must exist, but the proof is non-constructive, i.e. it doesn't actually show you how to get that object.

You're indeed correct that the backpropagation algorithm might not be able to find suitable weights. In fact, if you fix an architecture in advance, there might not even be any weights that would lead to a neural network that is a good approximation to $f$.

Why do we care?

Universality tells us, at least in principle, that we should be able to approximate the function we want, if we pick an appropriate architecture. The XOR Problem, for example, affects a type of model called a perceptron (essentially a single neuron neural network), and says that the XOR function cannot be approximated by perceptrons. It is always useful to know the kinds of functions that can and cannot be expressed by a certain type of model.

  • $\begingroup$ What was the usefulness of this article then? It sounds like he's saying functions can be approximated with sigmoid "towers" and "steps", but the neural network would have to have specific weights to force towers and steps, which is not even happening at all in the first place? $\endgroup$ – sangstar Apr 3 at 21:21
  • $\begingroup$ The point is that, if universal approximation wasn't true, there might be types of functions that you can never hope to represent with a neural network, because there simply isn't any set of weights which would do this. This would have practical implications because it would mean that no matter how much you train or look for a good architecture, you might still fail to find a good neural net for your problem. The whole section where he explicitly constructs weights is just to help you understand why the result is true. $\endgroup$ – htl Apr 4 at 8:21
  • $\begingroup$ So he gave an example of a possible way sigmoid functions can represent any function, but it isn't necessarily the way the neural network will use the sigmiods (by using stochastic gradient descent to get the best weights and biases), just a way that shows proof of concept. $\endgroup$ – sangstar Apr 4 at 16:37
  • $\begingroup$ That's correct, @sangstar. The theorem doesn't care what your algorithm is for finding the weights (and, conversely, makes no promises that backpropagation will find very good weights). It just says that there are some weights which would be a good fit: it doesn't tell you how to find them, and it doesn't say that backpropagation will converge to these weights: backprop isn't even mentioned once in the paper proving the theorem! $\endgroup$ – htl Apr 4 at 17:04
  • $\begingroup$ I suppose I find it counterintuitive, because the way the offer represents the “building blocks” it makes me feel like those “steps” and “towers” ought to be the orthonormal basis functions (because of how “fundamental” they look” that should form any functions, and that they should not be arbitrarily chosen but instead be the functions to use. Just an impression $\endgroup$ – sangstar Apr 4 at 17:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.