Do we know what the units of neural networks will do before we train them?

Question

I was learning about back-propagation and, looking at the algorithm, there is no particular 'partiality' given to any unit. What I mean by partiality there is that you have no particular characteristic associated with any unit, and this results in all units being equal in the eyes of the machine.

So, won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

I was reading a bit and watching few videos about backpropagation and, in the explanation given by Geoffrey Hinton, he talks about how we're trying to train the hidden units using the error derivatives w.r.t our hidden activities rather than using desired activities. This further strengthens my point about how by not adding any difference to the units, all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal.

Answer 1

In reverse order to how you asked:

all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal

This actually happens if you initialise the weights equally (e.g. all zero). Gradients in that case are the same to each neuron in the same layer, and everything changes in lockstep. A neural network without random weight initialisation will simply not work.

So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?

No, because random weight initialisation causes the gradients to be different, and the neuron activations will typically diverge to represent different "hidden" features that activate differently depending on the input.

What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine.

One interesting side-effect of this behaviour is that the "partiality" will often be assigned effectively randomly too, as the neural network will converge to features that somehow work. These features have no guarantee to be meaningful to a human being in the context of a problem to solve. They might be something that can be mapped to the problem, they might be some linear combination of something that can be understood, but often there is no obvious interpretation.