I've noticed there are two ways to interpret 0 and I'm a bit confused.

**Interpretation #1**

A 0 is just as meaningful as any non-zero number. Examples and reasoning:

- When we normalize images for input to a CNN we might divide by 255, subtract the mean, and divide by the standard deviation. There will be many pixels close to 0 but they are just as meaningful as other pixels in the image.

- We can do the classic digits MNIST exercise with white digits on a black background, or black digits on a white background. It doesn't matter.

- 0 shouldn't be special because the bias component of a neuron can always shift the 0-point.

**Interpretation #2**

A 0 or something close to 0 is a weakly activated neuron which therefore doesn't contribute much to what happens downstream in the network. Examples and reasoning:

- We use dropout to randomly set neurons to 0 as a means of regulation.

- The deeper layers in the forward pass of a well trained network tend to have a few strong activations with most activations being close to 0.

- When interpreting a CNN we can look at the last set of feature maps and observe strongly activated pixels. For instance [GradCAM](https://arxiv.org/pdf/1610.02391.pdf) relies on this idea to visualise which part of an image contributed to a prediction.