2
$\begingroup$

In a deep connected network, when every unit gets all the input features(X) so it has one parameter for every feature and every unit tweaks its parameters for loss optimization. What if we use only one unit and that one unit will have all the parameters which it can tweak for loss optimization. Is there a reason or benefit of using multiple units in every layer except the output layer?

$\endgroup$
1
$\begingroup$

So if I understand correctly, you're proposing to use a neutral net with $N$ input units (let's say data is in $\mathbf{R}^N$), 1 hidden unit, and whatever the necessary output needs to be.

Let's say we try to do this. Then each unit of the output layer is responsible for computing its output based on a single scalar input. So it's like as if you're saying the problem can be compressed to another problem that is 1-dimensional. Now let's say you're doing regression. By modeling your NN this way, you're saying there's a function that (approximately) maps a scalar (one dimensional object) to the target output. Suppose your output layer is a linear layer (which is usually the case in regression), this is saying that the dimension of the output space is at most 1. This is almost never the case, especially with complex data.

Now suppose you had several hidden units (call this number $M$) and the weights are initialized properly. Then, it's likely that your hidden unit will have dimension larger than one, perhaps even $M$, which allows for a richer output space as $M$ increases.

You can make similar arguments for the classification setting, but the output nonlinearity just makes it a little less clear.

$\endgroup$
2
  • $\begingroup$ Note that this does not answer the question "why... is not enough?", but it's just an attempt to interpret the situation. $\endgroup$
    – nbro
    Sep 25 '20 at 13:44
  • 1
    $\begingroup$ Sorry, about that, just added some elaboration. $\endgroup$
    – harwiltz
    Sep 25 '20 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.