This article attempts to provide a graphical justification of the universal approximation theorem.
It succeeds in showing that a linear combination of two sigmoids can produce essentially a bounded constant function or step function, and thus can therefore to a reasonable degree of approximation produce any function by essentially splitting up any function into a cluster (linear combination?) of these towers or steps.
However, he produced the steps and towers using specific weight parametrizations.
However, since when are we allowed to specify weights and biases? Isn't this all out of our hands and in the hands of cost function minimization?
I don't understand why he was dealing with setting weights to this, biases to that, when in my experience that is all done by "the machine" to minimize the cost function. I doubt the weights to minimize the cost function are arranged in the ways specified in order to form the towers and steps that were formed in this tutorial, so I kind of don't understand what all the hub-ub is all about.