Return to Answer

why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would make sense, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side...

Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? [...] why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would hold, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side (and thus broken the symmetry)...

It is called "localization" because it is localization ("look, here is the "dog" in the picture, not there").

It is not at all like that; positive means X and negative means not-X in the presence of class X (i.e. a specific X is present), and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would make sense, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero (we have actually chosen a side)...

It is called "localization" because it is localization, literally ("look, here is the "dog" in the picture, not there").

It is not at all like that; positive means X and negative means not-X in the presence of class X (i.e. a specific X is present), in the specific network, and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would make sense, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side...

A general remark, largely irrelevant to the exact issue: while your claim that

is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a trained network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases & weights matter, and we cannot transform them arbitrarily.

A general remark: while your claim that

is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a trained network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases & weights matter, and we cannot transform them arbitrarily.

UPDATE (after comments):

Are you pointing out that it's called "localization" and therefore must be so?

It is called "localization" because it is localization ("look, here is the "dog" in the picture, not there").

I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?"

It is not at all like that; positive means X and negative means not-X in the presence of class X (i.e. a specific X is present), and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.

why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would make sense, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero (we have actually chosen a side)...