4
$\begingroup$

I apologize if this is a repeated question or if this is too simple. I was learning about back-propagation and looking at the algorithm there is no particular 'partiality' given to any unit. What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine. So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

UPDATE: I was reading a bit and watching few videos about backpropagation and in the explanation given by Geoffrey Hinton, he talks about how we're trying to train the hidden units using the error derivatives w.r.t our hidden activities rather than using desired activites. This further strengthens my point about how by not adding any difference to the units, all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal.

$\endgroup$
8
$\begingroup$

In reverse order to how you asked:

all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal

This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural network without random weight initialisation will simply not work.

So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

No, because random weight initialisation causes the gradients to be different, and the neuron activations will typically diverge to represent different "hidden" features that activate differently depending on the input.

What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine.

One interesting side-effect of this behaviour is that the "partiality" will often be assigned effectively randomly too, as the neural network will converge to features that somehow work. These features have no guarantee to be meaningful to a human being in the context of a problem to solve. They might be something that can be mapped to the problem, they might be some linear combination of something that can be understood, but often there is no obvious interpretation.

$\endgroup$
  • $\begingroup$ Thanks. That answered my question perfectly. I guess I still didn't reach the part where the instructor mentioned the random initialization. $\endgroup$ – Htnamus Jul 5 '18 at 8:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.