A single neuron will be able to do linear separation. For example, XOR simulator network:

x1 --- n1.1
   \  /    \
    \/      \
    /\      /
   /  \    /
x2 --- n1.2 

Where x1, x2 are the 2 inputs, n1.1 and n1.2 are the 2 neurons in hidden layer, and n2.1 is the output neuron.

The output neuron n2.1 does a linear separation. How about the 2 neurons in hidden layer?

Is it still called linear separation (at 2 nodes and join the 2 separation lines)? or polynomial separation of degree 2?

I'm confused about how it's called because there are curvy lines in this wiki article: https://en.wikipedia.org/wiki/Overfitting

Some curvy separation?

Some other curvy separation?


I found out the curvy zigzag green line is not polynomial as if it were polynomial, a vertical line won't cut that curvy line more than 1 time.

It's the combination of straight lines (linear separation) of multiple neurons in the same layer. So it's linear separation ('linear' by previous_layer_output*weight, 'separation' by activation function), at multiple nodes.


What you have depicted is a nonlinear classificator. Although each stage does a linear separation, the sequential composition of linear separations is nonlinear. The nonlinearity of the neuron is key in this regard, as otherwise it would all be equivalent to a matrix multiplication, which is linear. You were right about the degree, although it's rarely called like that. People usually describe just the number of layers, and I guess the main reason is that it's not directly equivalent to such polynomials, as that depends on other factors (e.g. the activation function).

  • $\begingroup$ The ability to draw a hyperplane into the dataset is a property of the neural network itself. For example support vector machines are more powerful than normal perceptrons. It has nothing to do with the activation function because in the input layer there are no weights which can be adjusted. $\endgroup$ – Manuel Rodriguez Sep 11 '19 at 7:36
  • $\begingroup$ Each neuron draws its own hyperplane. On what facts do you base that a SVM is more powerful than a normal perceptron? What are you calling a "normal" perceptron? The activation function has everything to do, as otherwise the multiple weight matrices are equivalent to a single matrix, result of the product of all of them. $\endgroup$ – David Sep 11 '19 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.