# Is it really possible to create the "Perfect Cylinder" used in Universal Approximation Theorem for 1-hidden layer Neural Network?

There are proofs for the universal approximation theorem with just 1 hidden layer.

The proof goes like this:

1. Create a "bump" function using 2 neurons.

2. Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.

3. Decrease the step/radius to a very small value in order to approximate a cylinder. This is what I'm not convinced of

4. Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like this.

In this video, minute 42, the lecturer says

In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.

Here are the slides.

Here is a pdf version from another university, so you do not have to watch the video.

Why am I not convinced?

I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.

Now let's decrease the radius to 0.01:

Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.

Let's zoom in:

Let's decrease the radius to 0.0000001.

Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.

Python code to reproduce (requires NumPy and matplotlib): https://pastebin.com/CMXFXvNj.

So my questions are:

Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?

Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?

Both towers have the same shape

Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .

Vc = Volume Cylinder

Vt = Volume Tower

ratio(r) = Vc/Vt

What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.

So what they are saying is that: lim r -> 0 for ratio(r) = 1 But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.

Q3 Preface

An objection i got multiple times once from Dutta and once from D.W is that just decreasing the radious and plotting it isn't mathematical rigorous.

So let's assume in the limit of r=0 it's really a perfect cylinder.

One possible explanation for this would be that the limit is a special case and one can't approximate towards it

But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.

Q3 So why should we even care about this then ?

Further Clarifications

The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. I am NOT questioning the paper ! But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)

• What's your question? I don't see a question here. A question usually ends with a "?". You can't verify or disprove a statement about what happens "in the limit" by looking at finite instances.
– D.W.
Commented Jan 19, 2021 at 17:45
• There is a question mark in the title. Well if I decrease the width to a very small value the shape of the tower should change and get closer to a cylinder imo. If there doesn't happen any transformation of the shape there is no use for the universal approximation theorem. It's only useful if the tower gets closer to a cylinder when the step width decreases. If it's only a perfect cylinder when the step width is exactly zero, but a completly different shape when it's 0.0000000000001 above zero there is no use for it. Commented Jan 19, 2021 at 18:39
• Let me try to put it into mathematical terms as good as i can. Let;s define ts as the shape of the tower. ts depends on the number of neurons used, and on the step width(w). Let's ignore the neurons for now. So we have ts(w) . So you are saying that ts(w=0) is a perfect cylinder ? But ts(w=1) = ts(w=0.1) = ts(w=0.01) = ts(w=0.001) are all the same shape. So ts would be a discontinous function. Well, if this is really true, then what's the point of the Universal Approximation for a single hidden layer ? Commented Jan 19, 2021 at 19:01
• I suspect you've misunderstood the claim. I'll look forward to the revision.
– D.W.
Commented Jan 19, 2021 at 21:25
• @Just_a_fool Hey I did a lot of work related to this topic. I haven't had yet time to fully write up everything. But i am planning to do that in the future. I am going to have a look at your question soon. Commented Jan 10, 2022 at 20:05

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.

We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.

We can control the height of the tower with the weights.

First in 2d:

Unfortunately the closer we put this towers together the more they will start two influence each other.

But we can counter that with a negative tower between them.

Now in 3d:

This Answer is work in progress I will update it when i find out something new.

I think you misunderstood that part of the proof: you first need a limit on the number of neurons to get closer and closer to a cylinder. You are keeping them constant at 1000, thus indeed not getting any closer to the cylinder and exponential vanishing behavior.

Once you have the "epsilon-perfect" circles/cylinders, then you make them smaller and smaller, thus needing more and more copies of the "1-circle" setup.

My understanding is that this proof has those two numbers going to infinity: neurons-to-approximate-cylinder, and number-of-cylinders. You took into account the latter, but not the former.