I while ago I read that you can make subtle changes to an image that will ensure a good CNN will horribly misclassify the image. I believe the changes must exploit details of the CNN that will be used for classification. So we can trick a good CNN into classifying an image as a picture of a bicycle when any human would say it's an image of a dog. What do we call that technique, and is there an effort to make image classifiers robust against this trick?

  • $\begingroup$ This seems to be at least a partial duplicate of this: ai.stackexchange.com/q/15820/2444 (how can we generate inputs to an AI to fool them? is there any research on this?) or this ai.stackexchange.com/q/6800/2444 (in what ways could you fool an "AI", so not just a neural network, so this question seems to be too broad), or this ai.stackexchange.com/q/92/2444 (how is that possible that you can fool a NN), or this ai.stackexchange.com/q/6892/2444 (what tools are used to deal with AML). $\endgroup$
    – nbro
    Feb 18, 2021 at 19:43
  • 1
    $\begingroup$ However, currently, none of these seem to be exact duplicates, so I will leave this open for a while, because your question focuses on CNNs, and I hope that the answers below understand that. $\endgroup$
    – nbro
    Feb 18, 2021 at 20:07

2 Answers 2


These are known as adversarial attacks, and the specific examples that are misclassified are known as adversarial examples.

There is a reasonably large body of work on finding adversarial examples, and on making CNNs more robust (i.e. less prone to these attacks). An example is the DeepFool algorithm, which can be used to find perturbations of data which would cause the label to change.

There are several techniques in the literature which are used to fight against adversarial attacks. Here is a selection:

  1. Augmenting the training data with various random perturbations. This is intended to make the model more robust to the typical adversarial attack where random noise is added to an image. An example of this approach is discussed in [1].

  2. Constructing some sort of model to "denoise" input before feeding into the CNN. An example of this is Defense-GAN [2], which uses a generative adversarial model which models the "true" image distribution and finds an approximation of the input closer to the real distribution.


[1] Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. Explaining and Harnessing Adversarial Examples. ICLR (2015). URL.

[2] Pouya Samangouei, Maya Kabkab, Rama Chellappa. Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models. ICLR (2018). URL.


Those examples are called Adversarial Examples. I think it is important to understand why a CNN can be "tricked" like that:

We often expect human-like behavior when a model has a human-like performance. That is similar for CNNs. We expect they decide like we do, i.e. we look for the shape of objects.

However, as various experiments on common CNN show, that is not the case. A CNN looks for other features. It tries to minimize a loss function. And the fastest way to do this is often not the intended way. E.g. you think that a cow has like 4 legs and a certain shape of a head. A CNN might think that it is enough if there is something with 4 legs and a green background (because 9 of 10 images in the training dataset are like that). So for most of the cases the CNN does fine when it identifies a cow by these features.

Geirhos did an experiment in "Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness". He showed a CNN images that can be classified as A and B. E.g. like this image, taken from Geirhos paper above . In most of the cases a common CNN says this is not a cat. It is an elephant!

From Geirhos "Imagenet-trained cnns are biased towards texture;
increasing shape bias improves accuracy and robustness": CNNs favor texture information over shape information.

Or another similar experiment by Wang et al.: "High frequency component helps explain the generalization of convolutional neural networks". They revealed that common CNNs can classify images "correctly" although the low frequencies in that image (i.e. these are the frequencies that contain the shape information) are removed from the image.

In another experiment, Brendel & Bethge in "“Approximating cnns with bag-of-localfeatures models works surprisingly well on imagenet" redesigned CNNs so they can not use global shape information by their design. Their CNN performed also human-like. So why should other CNNs not use the features that were used by this network?

To come to the second part of your question: To make a CNN more robust you need to provide the CNN with images that contain only the features that you want it to learn. That is difficult with images. Removing texture information by applying style transfer is one option. Think about what you want the network to learn and what not. Then try to remove/hide/suppress what you do not want it to learn. Details are given in other answers.

There is also an approach called feature visualization that tries to generate visualizations of the features that are recognized by a CNN. However, the naive technique is almost useless, because these visualizations are often dominated by something like noise. (Except for a minimal part of a CNN.) This might indicate that CNNs do not focus on global shape information (as the visualizations do not sharp shapes, rather high-frequency textures).

Maybe this question helps a bit: How is it possible that deep neural networks are so easily fooled?


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .