Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

Question

There are proofs for the universal approximation theorem with just 1 hidden layer.

The proof goes like this:

1. Create a "bump" function using 2 neurons.

2. Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.

3. Decrease the step/radius to a very small value in order to approximate a cylinder. This is what I'm not convinced of

4. Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like this.

In this video, minute 42, the lecturer says

In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.

Here are the slides.

Here is a pdf version from another university, so you do not have to watch the video.

Why am I not convinced?

I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.

Let's start with a simple tower of radius 0.1:

Now let's decrease the radius to 0.01:

Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.

Let's zoom in:

Let's decrease the radius to 0.0000001.

Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.

Python code to reproduce (requires NumPy and matplotlib): https://pastebin.com/CMXFXvNj.

So my questions are:

Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?

Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?

Both towers have the same shape

Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .

Vc = Volume Cylinder

Vt = Volume Tower

ratio(r) = Vc/Vt

What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.

So what they are saying is that: lim r -> 0 for ratio(r) = 1 But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.

Q3 Preface

An objection i got multiple times once from Dutta and once from D.W is that just decreasing the radious and plotting it isn't mathematical rigorous.

So let's assume in the limit of r=0 it's really a perfect cylinder.

One possible explanation for this would be that the limit is a special case and one can't approximate towards it

But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.

Q3 So why should we even care about this then ?

Further Clarifications

The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. I am NOT questioning the paper ! But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)

Answer 1

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.

We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.

We can control the height of the tower with the weights.

First in 2d:

Unfortunately the closer we put this towers together the more they will start two influence each other.

But we can counter that with a negative tower between them.

Now in 3d:

This Answer is work in progress I will update it when i find out something new.

Answer 2

I think you misunderstood that part of the proof: you first need a limit on the number of neurons to get closer and closer to a cylinder. You are keeping them constant at 1000, thus indeed not getting any closer to the cylinder and exponential vanishing behavior.

Once you have the "epsilon-perfect" circles/cylinders, then you make them smaller and smaller, thus needing more and more copies of the "1-circle" setup.

My understanding is that this proof has those two numbers going to infinity: neurons-to-approximate-cylinder, and number-of-cylinders. You took into account the latter, but not the former.