Return to Answer

https://ai.meta.stackexchange.com/questions/1341/i-am-going-to-be-editing-the-old-questions-forward-any-opinions

Source Link

edit approved Apr 12, 2018 at 14:53

The convolutional networks work extremely well on this manifold, but basically, know nothing about the rest of the space. The classification of the points outside of the manifold is just a linear extrapolation based on the points inside the manifold. No wonder that some particular points are extrapolated incorrectly. The attacker only needs a way to navigate to the closest of these points.

Let me give you a concrete example how to fool a neural network. To make it compact, I'm going to use a very simple logistic regression network with one nonlinearity (sigmoid). It takes a 10-dimensional input x, computes a single number p=sigmoid(W.dot(x)), which is the probability of class 1 (versus class 0).

logistic regression

x and W

We'd like to find another example, y, which is very close to x, but is classified by the network as 1. Note that x is 10-dimensional, so we have the freedom to nudge 10 values, which is a lot.

x, W and y

If you look closely toat the previous example, you'll notice that I knew exactly how to tweak x in order to move it to the target class, because I knew the network gradient. What I did was actually a backpropagation, but with respect to the data, instead of weights.

The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request. It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easyeasily mess up with the results obtained for another network. The more networks are used, the more complex an attack becomes.

This technique allows to synthesizesynthesizing an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.

The convolutional networks work extremely well on this manifold, but basically know nothing about the rest of the space. The classification of the points outside of the manifold is just a linear extrapolation based on the points inside the manifold. No wonder that some particular points are extrapolated incorrectly. The attacker only needs a way to navigate to the closest of these points.

We'd like to find another example, y, which is very close to x, but is classified by the network as 1. Note that x is 10-dimensional, so we have freedom to nudge 10 values, which is a lot.

If you look closely to the previous example, you'll notice that I knew exactly how to tweak x in order to move it to the target class, because I knew the network gradient. What I did was actually a backpropagation, but with respect to the data, instead of weights.

The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request. It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easy mess up with the results obtained for another network. The more networks are used, the more complex an attack becomes.

This technique allows to synthesize an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.

logistic regression

x and W

We'd like to find another example, y, which is very close to x but is classified by the network as 1. Note that x is 10-dimensional, so we have the freedom to nudge 10 values, which is a lot.

x, W and y

If you look closely at the previous example, you'll notice that I knew exactly how to tweak x in order to move it to the target class, because I knew the network gradient. What I did was actually a backpropagation, but with respect to the data, instead of weights.

The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request. It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easily mess up with the results obtained for another network. The more networks are used, the more complex an attack becomes.

This technique allows synthesizing an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.

Source Link

created Sep 18, 2017 at 14:30

Maxim

All answers here are great, but, for some reason, nothing has been said so far on why this effect should not surprise you. I'll fill the blank.

Let me start with one requirement that is absolutely essential for this to work: the attacker must know neural network architecture (number of layers, size of each layer, etc). Moreover, in all cases that I examined myself, the attacker knows the snapshot of the model that is used in production, i.e. all weights. In other words, the "source code" of the network isn't a secret.

You can't fool a neural network if you treat it like a black box. And you can't reuse the same fooling image for different networks. In fact, you have to "train" the target network yourself, and here by training I mean to run forward and backprop passes, but specially crafted for another purpose.

Why is it working at all?

Now, here's the intuition. Images are very high dimensional: even the space of small 32x32 color images has 3 * 32 * 32 = 3072 dimensions. But the training data set is relatively small and contains real pictures, all of which have some structure and nice statistical properties (e.g. smoothness of color). So the training data set is located on a tiny manifold of this huge space of images.

Example

Suppose you know W=(-1, -1, 1, -1, 1, -1, 1, 1, -1, 1) and start with an input x=(2, -1, 3, -2, 2, 2, 1, -4, 5, 1). A forward pass gives sigmoid(W.dot(x))=0.0474 or 95% probability that x is class 0 example.

We'd like to find another example, y, which is very close to x, but is classified by the network as 1. Note that x is 10-dimensional, so we have freedom to nudge 10 values, which is a lot.

Since W[0]=-1 is negative, it's better for to have a small y[0] to make a total contribution of y[0]*W[0] small. Hence, let's make y[0]=x[0]-0.5=1.5. Likewise, W[2]=1 is positive, so it's better to increase y[2] to make y[2]*W[2] bigger: y[2]=x[2]+0.5=3.5. And so on.

The result is y=(1.5, -1.5, 3.5, -2.5, 2.5, 1.5, 1.5, -3.5, 4.5, 1.5), and sigmoid(W.dot(y))=0.88. With this one change we improved the class 1 probability from 5% to 88%!

Generalization

In general, the attacker starts with target distribution (0, 0, ..., 1, 0, ..., 0) (zero everywhere, except for the class it wants to achieve), backpropagates to the data and makes a tiny move in that direction. Network state is not updated.

Now it should be clear that it's a common feature of feed-forward networks that deal with a small data manifold, no matter how deep it is or the nature of data (image, audio, video or text).

Potection

The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request. It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easy mess up with the results obtained for another network. The more networks are used, the more complex an attack becomes.

Another possibility is to smooth the input before passing it to the network.

Positive use of the same idea

You shouldn't think that backpropagation to the image has only negative applications. A very similar technique, called deconvolution, is used for visualization and better understanding what neurons have learned.