It is okay to initialize the weights to zero for a simple logistic regression, but for a neural network to initialize the weights to parameters to all zero and then apply gradient descent, it won't work.
Assume here that you have in a 2-layered network with just 2 input features and in the hidden layer you have 2 nodes, and the weights are initialized as [u, v] for both the nodes.
So, while optimizing the weights, using gradient descent you would compute:
where α is the learning rate and J is the cost function that you want to minimize by optimizing the weights.
By kind of a proof by induction, you should be able to see that after every single iteration of training, your two hidden units are still computing exactly the same function because both hidden units start off computing the same function, have the same influence on the output unit. So even after multiple iterations, the two hidden units are still symmetric. Hence, there's really no point to having more than one hidden unit because they are all computing the same thing.
In summary, no matter how long you run gradient descent, both the two units compute exactly the same function which is not helpful, because you want the different hidden units to compute different functions. The solution to this is to initialize your parameters randomly.
For larger neural networks, say of three features and maybe a very large number of hidden units, a similar argument still works.
Note: It's okay to initialize the bias term to just zeros. Because so long as weights are initialized randomly, you start off with the different hidden units computing different things.