Why one unit in the layers of neural network is not enough?

Question

In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?

Answer 1

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.

Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.

Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.

You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.