Basic Functions and Results

Question

If the number of input neurons and output neurons doesn't change, what will change if I have one hidden layer, but first with 1 neuron, then with 4 neurons?

Taking into consideration the fact that each perceptron is able to linearly separate points on an unknown/unwritten linear function, would this then be able to, theoretically, instead of simply linearly separate points, separate points into those that occur inside a square, and those that occur outside?

This is, of course, without a bias neuron present.

Answer 1

If the number of input neurons and output neurons doesn't change, what will change if I have one hidden layer, but first with 1 neuron, then with 4 neurons?

I recommend that you try to implement a dense layer with Tensorflow (not using the Dense implementation, of course).

A dense layer is the full connection between to layers. What actually is learned/optimized in a neural network are the weights in between layers. Each weight is just a single floating point number. It's the connection between two nodes. If you increase the number of nodes of one layer, the two neighboring layers get more weights. Hence the optimization algorithm can adapt better.

For simplicity, I'll now assume you don't use biases.

If you have A input neurons and Z output neurons, a single hidden layer with 1 neutron means you have two matrices of shape Ax1 and 1xZ. if you change it to 4 hidden neurons the two matrices have the shape Ax4 and 4xZ