What is an adversarial attack?

Question

I'm reading this really interesting article CycleGAN, a Master of Steganography. I understand everything up until this paragraph:

we may view the CycleGAN training procedure as continually mounting an adversarial attack on $G$, by optimizing a generator $F$ to generate adversarial maps that force $G$ to produce a desired image. Since we have demonstrated that it is possible to generate these adversarial maps using gradient descent, it is nearly certain that the training procedure is also causing $F$ to generate these adversarial maps. As $G$ is also being optimized, however, $G$ may actually be seen as cooperating in this attack by learning to become increasingly susceptible to attacks. We observe that the magnitude of the difference $y^{*}-y_{0}$ necessary to generate a convincing adversarial example by Equation 3 decreases as the CycleGAN model trains, indicating cooperation of $G$ to support adversarial maps.

How is the CycleGAN training procedure an adversarial attack?

I don't really understand the quoted explanation.

Answer 1

Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.

Source: Attacking Machine Learning with Adversarial Examples

You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.

Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.

Answer 2

I guess they are talking about adversarial attacks in the same way that Szegedy et al. did in "Intriguing properties of neural networks"

They described "adversarial attacks" or "adversarial examples" as images with hardly perceptible perturbations that change the network's prediction.

For example, imagine you've trained a CNN to classify between a variety of classes. You take a picture of a dog $X_1$, and your CNN correctly classified it as a "dog", everything is fine so far.

Then you can add some small perturbation $p$ to your image $X_1$, so now you have a new image $X_2 = X_1 + p$. This new image still looks like a dog, because your perturbation was so small that is almost imperceptible.

The problem is that your CNN will classify your picture $X_2$ as something that is not a dog, for example, "fish".

Here, $X_2$ is an adversarial example created after using an adversarial perturbation $p$.

What is interesting about these adversarial perturbations $p$ is that they are not random. Actually, CNNs are very robust to random perturbations (noise), but adversarial perturbations $p$ are not like them. They are computed to fool a classifier (not only CNNs).

You can refer to figure 5 of the aforementioned paper for more examples.

Answer 3

Simple answer is tweaking an image in unnoticeable ways that completely fool software. Eg a cat that is identified as 99% likely "to be guacamole" https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program/#CU7dSAfQ5sqY