How is it possible that deep neural networks are so easily fooled?

Question

The following page/study demonstrates that the deep neural networks are easily fooled by giving high confidence predictions for unrecognisable images, e.g.

How this is possible? Can you please explain ideally in plain English?

Answer 1

First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network.

The deep neural network is the pre-trained network modeled on AlexNet provided by Caffe. To evolve images, both the directly encoded and indirectly encoded images, we use the Sferes evolutionary framework. The entire code base to conduct the evolutionary experiments can be download [sic] here. The code for the images produced by gradient ascent is available here.

Images that are actually random junk were correctly recognized as nothing meaningful:

In response to an unrecognizable image, the networks could have output a low confidence for each of the 1000 classes, instead of an extremely high confidence value for one of the classes. In fact, they do just that for randomly generated images (e.g. those in generation 0 of the evolutionary run)

The original goal of the researchers was to use the neural networks to automatically generate images that look like the real things (by getting the recognizer's feedback and trying to change the image to get a more confident result), but they ended up creating the above art. Notice how even in the static-like images there are little splotches - usually near the center - which, it's fair to say, are triggering the recognition.

We were not trying to produce adversarial, unrecognizable images. Instead, we were trying to produce recognizable images, but these unrecognizable images emerged.

Evidently, these images had just the right distinguishing features to match what the AI looked for in pictures. The "paddle" image does have a paddle-like shape, the "bagel" is round and the right color, the "projector" image is a camera-lens-like thing, the "computer keyboard" is a bunch of rectangles (like the individual keys), and the "chainlink fence" legitimately looks like a chain-link fence to me.

Figure 8. Evolving images to match DNN classes produces a tremendous diversity of images. Shown are images selected to showcase diversity from 5 evolutionary runs. The diversity suggests that the images are non-random, but that instead evolutions producing [sic] discriminative features of each target class.

Further reading: the original paper (large PDF)

Answer 2

The images that you provided may be unrecognizable for us. They are actually the images that we recognize but evolved using the Sferes evolutionary framework.

While these images are almost impossible for humans to label with anything but abstract arts, the Deep Neural Network will label them to be familiar objects with 99.99% confidence.

This result highlights differences between how DNNs and humans recognize objects. Images are either directly (or indirectly) encoded

According to this video

Changing an image originally correctly classified in a way imperceptible to humans can cause the cause DNN to classify it as something else.

In the image below the number at the bottom are the images are supposed to look like the digits But the network believes the images at the top (the one like white noise) are real digits with 99.99% certainty.

The main reason why these are easily fooled is that Deep Neural Network does not see the world in the same way as human vision. We use the whole image to identify things while DNN depends on the features. As long as DNN detects certain features, it will classify the image as a familiar object it has been trained on. The researchers proposed one way to prevent such fooling by adding the fooling images to the dataset in a new class and training DNN on the enlarged dataset. In the experiment, the confidence score decreases significantly for ImageNet AlexNet. It is not easy to fool the retrained DNN this time. But when the researchers applied such method to MNIST LeNet, evolution still produces many unrecognizable images with confidence scores of 99.99%.

More details here and here.

Answer 3

All answers here are great, but, for some reason, nothing has been said so far on why this effect should not surprise you. I'll fill the blank.

Let me start with one requirement that is absolutely essential for this to work: the attacker must know neural network architecture (number of layers, size of each layer, etc). Moreover, in all cases that I examined myself, the attacker knows the snapshot of the model that is used in production, i.e. all weights. In other words, the "source code" of the network isn't a secret.

You can't fool a neural network if you treat it like a black box. And you can't reuse the same fooling image for different networks. In fact, you have to "train" the target network yourself, and here by training I mean to run forward and backprop passes, but specially crafted for another purpose.

Why is it working at all?

Now, here's the intuition. Images are very high dimensional: even the space of small 32x32 color images has 3 * 32 * 32 = 3072 dimensions. But the training data set is relatively small and contains real pictures, all of which have some structure and nice statistical properties (e.g. smoothness of color). So the training data set is located on a tiny manifold of this huge space of images.

The convolutional networks work extremely well on this manifold, but basically, know nothing about the rest of the space. The classification of the points outside of the manifold is just a linear extrapolation based on the points inside the manifold. No wonder that some particular points are extrapolated incorrectly. The attacker only needs a way to navigate to the closest of these points.

Example

Let me give you a concrete example how to fool a neural network. To make it compact, I'm going to use a very simple logistic regression network with one nonlinearity (sigmoid). It takes a 10-dimensional input x, computes a single number p=sigmoid(W.dot(x)), which is the probability of class 1 (versus class 0).

Suppose you know W=(-1, -1, 1, -1, 1, -1, 1, 1, -1, 1) and start with an input x=(2, -1, 3, -2, 2, 2, 1, -4, 5, 1). A forward pass gives sigmoid(W.dot(x))=0.0474 or 95% probability that x is class 0 example.

We'd like to find another example, y, which is very close to x but is classified by the network as 1. Note that x is 10-dimensional, so we have the freedom to nudge 10 values, which is a lot.

Since W[0]=-1 is negative, it's better for to have a small y[0] to make a total contribution of y[0]*W[0] small. Hence, let's make y[0]=x[0]-0.5=1.5. Likewise, W[2]=1 is positive, so it's better to increase y[2] to make y[2]*W[2] bigger: y[2]=x[2]+0.5=3.5. And so on.

The result is y=(1.5, -1.5, 3.5, -2.5, 2.5, 1.5, 1.5, -3.5, 4.5, 1.5), and sigmoid(W.dot(y))=0.88. With this one change we improved the class 1 probability from 5% to 88%!

Generalization

If you look closely at the previous example, you'll notice that I knew exactly how to tweak x in order to move it to the target class, because I knew the network gradient. What I did was actually a backpropagation, but with respect to the data, instead of weights.

In general, the attacker starts with target distribution (0, 0, ..., 1, 0, ..., 0) (zero everywhere, except for the class it wants to achieve), backpropagates to the data and makes a tiny move in that direction. Network state is not updated.

Now it should be clear that it's a common feature of feed-forward networks that deal with a small data manifold, no matter how deep it is or the nature of data (image, audio, video or text).

Potection

The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request. It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easily mess up with the results obtained for another network. The more networks are used, the more complex an attack becomes.

Another possibility is to smooth the input before passing it to the network.

Positive use of the same idea

You shouldn't think that backpropagation to the image has only negative applications. A very similar technique, called deconvolution, is used for visualization and better understanding what neurons have learned.

This technique allows synthesizing an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.

Answer 4

An important question that does not yet have a satisfactory answer in neural network research is how DNNs come up with the predictions they offer. DNNs effectively work (though not exactly) by matching patches in the images to a "dictionary" of patches, one stored in each neuron (see the youtube cat paper). Thus, it may not have a high level view of the image since it only looks at patches, and images are usually downscaled to much lower resolution to obtain the results in current generation systems. Methods which look at how the components of the image interact may be able to avoid these problems.

Some questions to ask for this work are: How confident were the networks when they made these predictions? How much volume do such adversarial images occupy in the space of all images?

Some work I am aware of in this regard comes from Dhruv Batra and Devi Parikh's Lab at Virginia Tech who look into this for question answering systems: Analyzing the Behavior of Visual Question Answering Models and Interpreting Visual Question Answering models.

More such work is needed, and just as the human visual system does also get fooled by such "optical illusions", these problems may be unavoidable if we use DNNs, though AFAIK nothing is yet known either way, theoretically or empirically.

Answer 5

How is it possible that deep neural networks are so easily fooled?

Deep neural networks are easily fooled by giving high confidence predictions for unrecognizable images. How is this possible? Can you please explain ideally in plain English?

Intuitively, extra hidden layers ought to make the network able to learn more complex classification functions, and thus do a better job classifying. While it may be named deep learning it's actually a shallow understanding.

Test your own knowledge: Which animal in the grid below is a Felis silvestris catus, take your time and no cheating. Here's a hint: which is a domestic house cat?

For a better understanding checkout: "Adversarial Attack to Vulnerable Visualizations" and "Why are deep neural networks hard to train?".

The problem is analogous to aliasing, an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled, and the stagecoach-wheel effect, where a spoked wheel appears to rotate differently from its true rotation.

The neural network doesn't know what it's looking at or which way it's going.

Deep neural networks aren't an expert on something, they are trained to decide mathematically that some goal has been met, if they are not trained to reject wrong answers they don't have a concept of what is wrong; they only know what is correct and what is not correct - wrong and "not correct" are not necessarily the same thing, neither is "correct" and true.

The neural network doesn't know right from wrong.

Just like most people wouldn't know a house cat if they saw one, two or more, or none. How many house cats in the above photo grid, none. Any accusations of including cute cat pictures is unfounded, those are all dangerous wild animals.

Here's another example. Does answering the question make Bart and Lisa smarter, does the person they are asking even know, are there unknown variables that can come into play?

We aren't there yet but neural networks can quickly provide an answer that is likely to be correct, especially if it was properly trained to avoid all misteps.

Answer 6

Can't comment(due to that required 50 rep), but I wanted to make a response to Vishnu JK and the OP. I think you guys are skipping the fact that the neural network only really is saying truly from a programmatic standpoint that "this is most like".

For example, while we can list the above image examples as "abstract art", they definitively are most like was is listed. Remember learning algorithms have a scope on what they recognize as an object and if you look at all the above examples... and think about the scope of the algorithum... these make sense (even the ones at a glance we would recognize as white noise). In Vishnu example of the numbers, if you fuzz your eyes and bring the images out of focus, you can actually in every case spot patterns that really closely reflect the numbers in question.

The problem that is being shown here is that the algorithm appears to not have a "unknown case". Basically when the pattern recognition says that it doesn't exist in the output scope. (so a final output node group that says this is nothing that I know off). For example, people do this as well, as it's one thing humans and learning algorithms have in common. Here's a link to show what I'm talking about (what is the following, define it) using only known animals that exist:

Now as a person, limited by what I know and can say, I'd have to conclude that the following is an elephant. But it's not. Learning algorithms (for the most part) do not have a "like a" statement, the out put always validates down to a confidence percentage. So tricking one in this fashion is not surprising... what is of course surprising is that based on it's knowledge set, it actually comes to the point in which, if you look at the above cases listed by OP and Vishnu that a person... with a little looking... can see how the learning algorithm probable made the association.

So, I wouldn't really call it a mislabel on the part of the algorithm, or even call it a case where it's been tricked... rather a case where it's scope was developed incorrectly.

Answer 7

The neural networks can be easily fooled or hacked by adding certain structured noise in image space (Szegedy 2013, Nguyen 2014) due to ignoring non-discriminative information in their input.

For example:

Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.²⁰¹⁵

So basically the high confidence prediction in certain models exists due to a 'combination of their locally linear nature and high-dimensional input space'.²⁰¹⁵

Published as a conference paper at ICLR 2015 (work by Dai) suggest that transferring discriminatively trained parameters to generative models, could be a great area for further improvements.

Answer 8

There is already many good answers, I will just add to those that came before mine:

This type of images you are referring to are called adversarial perturbations, (see 1, and it is not limited to images, it has been shown to apply to text too, see Jia & Liang, EMNLP 2017. In text, the introduction of an irrelevant sentence which doesn't contradict the paragraph has been seen to cause the network to come to a completely different answer (see see Jia & Liang, EMNLP 2017).

The reason they work is due to the fact that neural network view images in a different way from us, coupled with the high dimentionality of the problem space. Where we see the whole picture, they see a combination of features which combine to form an object(Moosavi-Dezfooli et al., CVPR 2017). According to perturbation generated against one network has been seen to have high likelihood to work on other networks:

In the figure above, it is seen that The universal perturbations computed for the VGG-19 network, for example, have a fooling ratio above 53% for all other tested architectures.

So how do you deal with the threat of adversarial perturbations? Well, for one, you can try to generate as many perturbations as you can and use them to fine-tune your model. Whist this somewhat solves the problem, it doesn't solve the problem entirely. In (Moosavi-Dezfooli et al., CVPR 2017) the author reported that, repeating the process by computing new perturbations and then fine-tuning again seems to yield no further improvements, regardless of the number of iterations, with the fooling ratio hovering around 80%.

Perturbations are an indication of the shallow pattern matching that neural networks perform, coupled with their minimal lack of in-depth understanding of the problem at hand. More work still needs to be done.

Answer 9

Neural networks are easily fooled, provided you know how to fool them.

Consider a linear network with an input layer and an output layer, which has an error function E (we don't need hidden layers to show how to fool a network). For a given input image x, E measures the (squared) difference between the network's output y and the desired (correct) output.

The output unit’s state y is given by the inner product of x with the output unit’s weight vector w,so that

y=w·x. 

If we change x to x′ by adding ∆x then the output will change by ∆y to

y′ = w·x+w·∆x (9.4) 

   = y + ∆y. (9.5)

Notice that ∆x defines a direction in the input space; the question is, which direction ∆x will have most impact on y?

By definition, a change in x in the direction ∇E produces the largest possible change in y. An adversarial image x' is constructed by taking the derivative ∇E of E with respect to the input image x, that is,

   x′ = x + ε∇E,

where ε is a small constant. By definition, ∇E is the direction of steepest ascent, so the modification ε∇E to x will alter y more than a change in any other direction.

This is an extract from the book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning (2019).