4
$\begingroup$

There are proofs for the universal approximation theorem with just 1 hidden layer.

The proof goes like this:

  1. Create a "bump" function using 2 neurons.

  2. Create (infinitely) many of these step functions with different angles in order to create a tower-like shape.

  3. Decrease the step/radius to a very small value in order to approximate a cylinder. This is what I'm not convinced of

  4. Using these cylinders one can approximate any shape. (At this point it's basically just a packing problem like this.

In this video, minute 42, the lecturer says

In the limit that's going to be a perfect cylinder. If the cylinder is small enough. It's gonna be a perfect cylinder. Right ? I have control over the radius.

Here are the slides.

enter image description here

Here is a pdf version from another university, so you do not have to watch the video.

Why am I not convinced?

I created a program to plot this, and even if I decrease the radius by orders of magnitude it still has the same shape.

Let's start with a simple tower of radius 0.1:

enter image description here

Now let's decrease the radius to 0.01:

enter image description here

Now, you might think that it gets close to a cylinder, but it just looks like it is approximating a perfect cylinder, because of the zoomed out effect.

Let's zoom in:

enter image description here

Let's decrease the radius to 0.0000001.

enter image description here

Still not a perfect cylinder. In fact, the "quality" of the cylinder is the same.

Python code to reproduce (requires NumPy and matplotlib): https://pastebin.com/CMXFXvNj.

So my questions are:

Q1 Is it true that we can get a perfect cylinder solely by decreasing the radius of the tower to 0 ?

Q2 If this true, why is there no difference when I plot it with different radii(0.1, vs 1e-7) ?

Both towers have the same shape

Clarification: What do I mean with: same shape ? Let's say we calculate the volume of an actual cylinder(Vc) with the same raius and height as our tower and divide it by the volume of the tower(Vt) .

Vc = Volume Cylinder

Vt = Volume Tower

ratio(r) = Vc/Vt

What this documents/lectures claim that is the ratio of these 2 volumes depends on the radius but in my view it's just constant.

So what they are saying is that: lim r -> 0 for ratio(r) = 1 But my experiments show that: lim r -> 0 for ratio(r) = const and don't depend on the radius at all.

Q3 Preface

An objection i got multiple times once from Dutta and once from D.W is that just decreasing the radious and plotting it isn't mathematical rigorous.

So let's assume in the limit of r=0 it's really a perfect cylinder.

One possible explanation for this would be that the limit is a special case and one can't approximate towards it

But if that is true this would imply that there is no use for it since it's impossible to have a radius of exactly zero. It would only be useful if we could get gradually closer to a perfect cylinder by decreasing the radius.

Q3 So why should we even care about this then ?

Further Clarifications

The original universal approximation theorem proof for single hidden layer neural networks was done by G. Cybenko. Then I think people tried to make some visual explations for it. I am NOT questioning the paper ! But i am questioning the visual explanation given in the linked lecutre/pdf (made by other people)

$\endgroup$
13
  • 1
    $\begingroup$ What's your question? I don't see a question here. A question usually ends with a "?". You can't verify or disprove a statement about what happens "in the limit" by looking at finite instances. $\endgroup$ – D.W. Jan 19 at 17:45
  • $\begingroup$ There is a question mark in the title. Well if I decrease the width to a very small value the shape of the tower should change and get closer to a cylinder imo. If there doesn't happen any transformation of the shape there is no use for the universal approximation theorem. It's only useful if the tower gets closer to a cylinder when the step width decreases. If it's only a perfect cylinder when the step width is exactly zero, but a completly different shape when it's 0.0000000000001 above zero there is no use for it. $\endgroup$ – KoKlA Jan 19 at 18:39
  • $\begingroup$ Let me try to put it into mathematical terms as good as i can. Let;s define ts as the shape of the tower. ts depends on the number of neurons used, and on the step width(w). Let's ignore the neurons for now. So we have ts(w) . So you are saying that ts(w=0) is a perfect cylinder ? But ts(w=1) = ts(w=0.1) = ts(w=0.01) = ts(w=0.001) are all the same shape. So ts would be a discontinous function. Well, if this is really true, then what's the point of the Universal Approximation for a single hidden layer ? $\endgroup$ – KoKlA Jan 19 at 19:01
  • $\begingroup$ Could you edit the question to include a self-contained statement of the theorem, and then tell us what kind of answer you are looking for? I don't really want to watch an external video to understand what you are asking; and we discourage questions that rely on us to click external links to understand what is being asked. Slides and informal statements like "nearly perfect cylinder" are not a substitute for an actual mathematical exposition with a precise statement of the theorem. $\endgroup$ – D.W. Jan 19 at 19:57
  • $\begingroup$ It seems like the fact that you have to zoom in farther and farther to see the inaccuracy in the approximation is an indication that there is a sense in which it is becoming a closer approximation to a perfect cylinder. No, it will never be a perfect cylinder for any finite value, but that is not a contradiction. I am not sure what kind of answer you are looking for, though, or exactly what the question is. $\endgroup$ – D.W. Jan 19 at 20:00
1
$\begingroup$

The more I think about it the more convinced I am that the visual explanation from the linked lecture is wrong. But the good news is there are still some ways to get close to the cylinder but not before the activation of the last neuron but instead afterwards. I haven't done it with simgoid. But I tried with ReLu instead for now.

We can cut the tower at the very top (thanks to ReLu and a bias). The closer to the top we cut it the more it will be like a cylinder.

We can control the height of the tower with the weights.

First in 2d:

enter image description here

Unfortunately the closer we put this towers together the more they will start two influence each other.

enter image description here

But we can counter that with a negative tower between them. enter image description here

Now in 3d:

enter image description here

This Answer is work in progress I will update it when i find out something new.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.