I'm learning about multilayer perceptrons, and I have a quick theory question in regards to hidden layer neurons.

I know we can use two hidden layers to solve a non-linearly seperable problem by allowing for a representation with two linear seperators. However, we can solve a non-linearly seperable problem using only one hidden layer.

This seems fine, but what kind of representation does one hidden layer add? My question is how is the dimensionality of the output affected?

I've drawn a diagram of a multilayer perceptron with one hidden layer neuron. I used this same layout to solve a non-linearly seperable problem. The single hidden layer node is inside the red square. Forgive my poor MS-Paint skills.

simple perceptron

  • 1
    $\begingroup$ can you please label the geometrics shape ? (what is a square, circle and so on). And I think you misunderstand some term here. A perceptron don't have and hidden layer. A perceptron is only one 'cell' or 'neuron' with activation. Maybe you want to say that you made a diagram of a Multi Layer Perceptron ? $\endgroup$ – Jérémy Blain Oct 8 '18 at 15:20
  • $\begingroup$ my apologies, got a little ahead of myself there. I've added an edit to clarify $\endgroup$ – Howard P Oct 8 '18 at 15:25

When we add a single layer with a non-linear activation function, right after the application of activation function, a new basis function is found(for that neuron) which is some combination of weights and biases, which acts as a new way to view or analyse the feature sets. With increasingly deep network, we keep finding representations which are new basis vectors of previous layer weight and bias combinations, that is higher level representation. If they're error free, they'll give better performances, but if small errors creep in earlier basis vectors, error increases through depth, which is the problem of overfitting.

A nice analogy is Taylor series where 1,x,x^2 and so on are the basis vectors for estimating the function in 1D. With deep networks, imagination is out of the box and you can analyse them mathematically, atleast that's my approach now.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.