Can a NN be configured to indicate which points of the input influenced its prediction and how?

Question

Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.

I suppose a good model would predict a 6, but maybe with less than 100% certainty (or maybe with 100% certainty - I don't know that it matters for the purpose of this question).

Is there any way to get information about which pixels most strongly influenced the decision of the CNN, and which pixels were not so important? So to represent that visually, green means that those pixels were important:

Or conversely, is it possible to highlight pixels which did not contribute to the outcome (or which cast doubt on the outcome thereby reducing the certainty from 100%)

Answer 1

Yes there definitely is, and research into this has actually resulted in some really cool behaviour.

One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.

Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).

You can also determine what each filter in a convolutional layer is looking at using similar techniques.

It's a super interesting field of machine learning, and I highly recommend taking a look at this lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.

Answer 2

There are many frameworks which allow you to do that.

One of them, which supports many different techniques for visualization, can be found here: https://github.com/marcoancona/DeepExplain