My knowledge
Suppose you have a layer that is fully connected, and that each neuron performs an operation like
a = g(w^T * x + b)
were a
is the output of the neuron, x
the input, g
our generic activation function, and finally w
and b
our parameters.
If both w
and b
are initialized with all elements equal to each other, then a
is equal for each unit of that layer.
This means that we have symmetry, thus at each iteration of whichever algorithm we choose to update our parameters, they will update in the same way, thus there is no need for multiple units since they all behave as a single one.
In order to break the symmetry, we could randomly initialize the matrix w
and initialize b
to zero (this is the setup that I've seen more often). This way a
is different for each unit so that all neurons behave differently.
Of course, randomly initializing both w
and b
would be also okay even if not necessary.
Question
Is randomly initializing w
the only choice? Could we randomly initialize b
instead of w
in order to break the symmetry? Is the answer dependent on the choice of the activation function and/or the cost function?
My thinking is that we could break the symmetry by randomly initializing b
, since in this way a
would be different for each unit and, since in the backward propagation the derivatives of both w
and b
depend on a
(at least this should be true for all the activation functions that I have seen so far), each unit would behave differently. Obviously, this is only a thought, and I'm not sure that is absolutely true.