I've noticed there are two ways to interpret 0. They are seemingly at odds with each other so my question is around how to reconcile these two viewpoints.

Interpretation #1

A 0 is just as meaningful as any non-zero number. Examples and reasoning:

• When we normalize images for input to a CNN we might divide by 255, subtract the mean, and divide by the standard deviation. There will be many pixels close to 0 but they are just as meaningful as other pixels in the image.

• We can do the classic digits MNIST exercise with white digits on a black background, or black digits on a white background. It doesn't matter.

• 0 shouldn't be special because the bias component of a neuron can always shift the 0-point.

Interpretation #2

A 0 or something close to 0 is a weakly activated neuron which therefore doesn't contribute much to what happens downstream in the network. Examples and reasoning:

• We use dropout to randomly set neurons to 0 as a means of regulation.

• The deeper layers in the forward pass of a well trained network tend to have a few strong activations with most activations being close to 0.

• When interpreting a CNN we can look at the last set of feature maps and observe strongly activated pixels. For instance GradCAM relies on this idea to visualise which part of an image contributed to a prediction.