I apologize if this is a repeated question or if this is too simple. I was learning about back-propagation and looking at the algorithm there is no particular 'partiality' given to any unit. What I mean by partiality there is that, you have no particular characteristic associated with any unit and this results in all units being equal in the eyes of the machine. So won't this result in the same activation values of all the units in the same layer? Won't this lack of 'partiality' render neural networks obsolete?
UPDATE: I was reading a bit and watching few videos about backpropagation and in the explanation given by Geoffrey Hinton, he talks about how we're trying to train the hidden units using the error derivatives w.r.t our hidden activities rather than using desired activites. This further strengthens my point about how by not adding any difference to the units, all units in a layer become equal since initially the errors due to all of them are the same and thus we train them to be equal.