In adversarial machine learning, how does an attacker have access to the test and training dataset in order to poison it?

Question

In the field of adversarial machine learning, machine learning models are vulnerable to attacks both on the test and training data set. However, how does the attacker get access to these datasets? How do these datasets get manipulated/tampered with?

Answer 1

They don't have acces to the original training or test dataset. Machine learning environments are build on the premise of a benign environment. The models are trained on real data (real inputs). When someone sends a made up input (fake input) it is very easy to fool the model.

This is used for example in image recognition. Imagine a fotograph of a panda. the model may correctly identify this fotograph as a panda. With knowledge of the model you can now alter some pixels in the fotograph. To the human Eye, the fotographs will appear exactly the same, but the model can be fooled to believe the fotograph is actually of a gibbon.

This is all done after the training of the model and doesn't require the original datasets.

For more info, visit this site: https://medium.com/@ml.at.berkeley/tricking-neural-networks-create-your-own-adversarial-examples-a61eb7620fd8

Answer 2

In adverserial machine learning, someone (program or human) attempts to fool an existing model with a malicious input.

The best human example would be an optical illusion. The human brain's model for image processing starts outputting wrong information when looking at an optical illusion. So in the end we see wrong colour, shape, etc. In this case, the optical illusion would be considered as the malicious input.

We can trick the human brain’s model through images created with trial and error.

So, if you just have the trained model at hand, you don’t have to know the data it has been trained with. You just need to be able to input a value to the model and get the output.

Answer 3

We can manipulate a model's test data set if the machine learning model takes user input and uses it to resample test data set. The actual training dataset of the ML model does not get manipulated, but if we figure out the ML model through an exploratory attack (sending a lot of inquiries to the ML model to find out its nature), we can generate a training dataset which was built into the original ML model.