2
$\begingroup$

Suppose I want to classify a dataset like the MNIST handwritten dataset, but it has added distractions. For example, here we have a 6 but with extra strokes around it that don't add value.

enter image description here

I suppose a good model would predict a 6, but maybe with less than 100% certainty (or maybe with 100% certainty - I don't know that it matters for the purpose of this question).

Is there any way to get information about which pixels most strongly influenced the decision of the CNN, and which pixels were not so important? So to represent that visually, green means that those pixels were important:

enter image description here

Or conversely, is it possible to highlight pixels which did not contribute to the outcome (or which cast doubt on the outcome thereby reducing the certainty from 100%)

enter image description here

$\endgroup$
1
$\begingroup$

Yes there definitely is, and research into this has actually resulted in some really cool behaviour.

One of the simplest ways is to simply back propagate the gradient all the way back to the input. Areas of the input that affected the final decision will receive larger gradients. Interestingly, this also sort of works as a rudimentary form of semantic segmentation.

Other ways are to change segments of the input and to see how that affects the output (like add 0.1 to a pixel).

You can also determine what each filter in a convolutional layer is looking at using similar techniques.

It's a super interesting field of machine learning, and I highly recommend taking a look at this lecture that is completely free and one of the most interesting ones I've personally seen. it will explain all this much better than I have done.

| improve this answer | |
$\endgroup$
1
$\begingroup$

There are many frameworks which allow you to do that.

One of them, which supports many different techniques for visualization, can be found here: https://github.com/marcoancona/DeepExplain

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.