How are non-linear surfaces formed in the training of a neural network? [closed]

Question

Desperate trying to understand something for couple of weeks. All those questions are actually one big question.Please help me. Time-codes and screens in my question refer to this great(IMHO) 3d explanation:

https://www.youtube.com/watch?v=UojVVG4PAG0&list=PLVZqlMpoM6kaJX_2lLKjEhWI0NlqHfqzp&index=2

.... Here is the case: Say I have 2 inputs (lets call them X1 and X2) into my ANN. Say X1= persons age and X2=years of education.

1) First question: do I plug those numbers as is or normalize them 0-1 as a "preprocessing"?

2) As I have 2 weights and 1 bias, actually I am going to plug my inputs to X1*W1+X2*W2=output formula. This is 2d plane in a 3d space if I am not mistaken(time-code 5:31):

Thus when I plug in my variables, like in regression I will get on a Z axis my output. So the second question is: am I right up to here?

-----------------From here come real important couple of questions.

3) My output (before I plug it into the activation function) is just a simple number, IT IS NOT A PLANE and NOT A SURFACE, but a simple scalar, without any sigh on it coming from 2d surface in a 3d space(though it does come from there). Thus, when I plug this number (which was Z value in a previous step) into the activation function (say sigmoid) my number enters there in to the X axis, and we get as an output some Y value. As I understand this operation was totally 2d operation, is was 2d sigmoid and not some kind of 3dsigmoidal surface.

So here is the question: If I am right, why do we see in this movie (and couple of other places) such an explanation? (time-code 12:55):

4)Now lets say that I was right in the previous step and as an output from the activation function I do get a simple number not a 2d surface and not a 3d one. I just have some number like I had in the very beginning of the ANN as an input (age, education etc). If i want to add another layer of neurons, this very number enters there as is not telling any one the "secret" that it was created by some kind of sigmoid. In this next layer this number is about to take similar transformations as it happened to age and education in a previous layer, it is going to be Xn in just the same scenario: sigmoid(XnWn+XmWm=output) and in the end we will get once again just a number. If I am right, why in the movie they say (time-code 14:50 ) that when we add together two activation functions we get something unlinear. They show result of such "addition" first as 2d (time-code 14:50 and 14:58). So, Here comes my question: how come that they "add" two activation functions, if to the second activation function reaches just a simple number as said above he is not telling any one the "secret" that it was created by some kind of sigmoid.

5) And then again, they show this addition of 3d surfaces (time-code 19:39 )
How it is possible? I mean again there should not happen any addition of surfaces, because no surface passes to next step but a number. What do I miss?

Answer 1

Hi and welcome to the community. It's important to understand these basic concepts very clearly.

You have to first understand the basic unit of a neural network, a single node/neuron/perceptron. Let us forget all about Neural Networks for a bit, and talk about something far simpler.

Linear Regression

In the above figure, we clearly have one independent variable on the x-axis, and one dependent variable on the y-axis. The red line has an intercept of zero, and let's say a slope of 0.5. Therefore, $$ y = 0.5x + 0 $$

This, right here, is a single perceptron. You take a value of x, lets say 8, pass it through the the node, and get a value as output, 4. Simple! But what is the model in this case? Is it the output? No. Its the set [0.5, 0] that represents the red line above. The outputs are simply points on that line.

A neural network model is always a set of values - a matrix or a tensor, if you will.

The plots in your question, do not represent outputs. They represent the models. But now that you've possibly understood what a linear model with one indpendent variable looks like, I hope you can appreciate that having 2 independent variables will give us a plane in 3-D space. This is called multiple regression.

This forms the first layer of a neural network with linear activation functions. Assuming $ x_{i} $ and $ x_{j} $ as the two independent variables, the first layer computes $$ y_{1} = w_{1}x_{i} + w_{2}x_{j} + b_{1} $$

Note that while $ y_{1} $ is the output of the first layer, the set $ [w_{1}, w_{2}, b_{1}] $ is the model of the first layer and can be plotted as a plane in 3D space. The second layer, again a linear layer, computes $$ y_{2} = w_{3}y_{1} + b_{2} $$

Substitute $ y_{1} $ in above and what do you get? Another linear model!

$$ y_{2} = w_{3}(w_{1}x_{i} + w_{2}x_{j} + b_{1}) + b_{2} $$

Adding layers to a neural network is only compounding of functions.

Compounding linear functions on linear functions result in linear functions.

Well, then, what was the point of adding a layer? Seems useless, right?

Yes, adding linear layers to a neural network is absolutely useless. But what happens if the activation functions of each perceptron, each layer was not linear? For example the sigmoid or the most widely used today, ReLU.

Compounding non-linear functions on non-linear functions can increase non-linearity.

The ReLU looks like this $$ y = max(0, x) $$

This is definitely non-linear but not as non-linear as let's say the sine wave. But can we approximate the sine wave by somehow "compounding" multiple, say $ N $ ReLUs?

$$ \sin(x) \approx a + \sum_{N}b*max(0, c + dx)$$

And here the variables $ a, b,c, d $ are the trainable "weights" in neural network terminology.

If you remember the structure of the perceptron, the first operation is often denoted as a summation over all the inputs. This is how non-linearity is approximated in Neural Networks. Now one may ask: So, summing over non-linear functions can approximate any function, right? So a single hidden layer between input layer and output layer (one that sums over all the outputs of the hidden layer units) should be enough? Why do we often see neural network architectures with so many hidden layers? This is one of the most important yet often overlooked aspect of neural networks and deep-learning.

To quote, Dr. Ian J Goodfellow, one of the brightest minds in AI,

A feedforward network with a single (hidden) layer is sufficient to represent any function, but the layer may be infeasibly large and may fail to learn and generalize correctly.

So, what is the ideal number of hidden layers? There's no magic number! ;-)

For more mathematical rigor on how neural networks approximate non-linear functions, one should learn about the Universal Approximation Theorem. Beginners should check this out.

But why should we care for increased non-linearity? For that I'd direct you to this.

Note that all of the above discussion is with respect to regression. For classification, the non-linear surface learned is regarded as a decision boundary and points above and below the surface are classified into different classes. However, an alternative, and arguably better, way to look at this is that given a dataset that is not linearly seperable, a neural network first transforms the input dataset into a linearly seperable form and then uses a linear decision boundary on it. For more on this, definitely check out Christopher Olah's amazing blog.

Finally, yes all independent variables must be normalized before training a neural network. This is to equalize the scale of different variables. More info here.

Answer 2

The output of any node is simply a scalar number. For a given input you get a specific scalar output. What is being shown is the surfaces that get generated as you VARY x1 and x2 over their input range. To answer your first question it is always best to scale your inputs.