I've noticed there are two ways to interpret 0. They are seemingly at odds with each other so my question is around how to reconcile these two viewpoints. And if both interpretations are correct depending on context, how so?

Interpretation #1

A 0 is just as meaningful as any non-zero number. Examples and reasoning:

• When we normalize images for input to a CNN we might divide by 255, subtract the mean, and divide by the standard deviation. There will be many pixels close to 0 but they are just as meaningful as other pixels in the image.

• We can do the classic digits MNIST exercise with white digits on a black background, or black digits on a white background. It doesn't matter.

• 0 shouldn't be special because the bias component of a neuron can always shift the 0-point.

Interpretation #2

A 0 or something close to 0 is a weakly activated neuron which therefore doesn't contribute much to what happens downstream in the network. Examples and reasoning:

• We use dropout to randomly set neurons to 0 as a means of regulation.

• The deeper layers in the forward pass of a well trained network tend to have a few strong activations with most activations being close to 0.

• When interpreting a CNN we can look at the last set of feature maps and observe strongly activated pixels. For instance GradCAM relies on this idea to visualise which part of an image contributed to a prediction.

EDIT

I want to clarify my question by making it a little less philosophical. Here are some practical reasons I might care about the answer:

• Under Interpretation 2, I sometimes worry about the way I normalize my inputs. If 0 has significance, then maybe I care about which parts of my input image are close to 0, or whether or not I have white digits on a black background vs black digits on a white background. Should it matter?

• Under Interpretation 1, the GradCAM technique I mentioned above doesn't seem so sensible. Without understanding the answer to my main question, I can't really understand how GradCAM works.