Yes, if the activation function of the network is **not zero centered**, $y = f(x^{T}w)$ is always *positive* or always *negative*. Thus, the output of a layer is always being moved to either the positive values or the negative values. As a result, the **weight vector** needs **more updates** to be trained properly, and the number of **epochs** needed for the network to get trained also **increases**. This is why the **zero centered property is important**, though it is **NOT** necessary.

**Zero-centered activation functions** ensure that the **mean activation value** is around **zero**. This property is important in deep learning because it has been empirically shown that models operating on **normalized** data––whether it be inputs or latent activations––enjoy *faster convergence*. 

Unfortunately, zero-centered activation functions like `tanh` **saturate** at their **asymptotes** –– the gradients within this region get **vanishingly smaller** over time, leading to a weak training signal.

`ReLU` avoids this problem but it is not zero-centered. Therefore all-positive or all-negative activation functions whether `sigmoid` or `ReLU` can be difficult for **gradient-based** optimization. So, To solve this problem deep learning practitioners have invented a myriad of *Normalization* layers (**batch norm, layer norm, weight norm**, etc.). we can **normalize the data in advance to be zero-centered** as in batch/layer normalization.


**Reference:**

[*A Survey on Activation Functions and their relation with Xavier and He Normal Initialization*][1]





  [1]: https://arxiv.org/abs/2004.06632