31
$\begingroup$

The paper The Limitations of Deep Learning in Adversarial Settings explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. The authors experiment with a neural network meant to read handwritten digits, undermining its reading ability by distorting the samples of handwritten digits that the neural network is trained with.

I'm concerned that malicious actors might try hacking AI. For example

  • Fooling autonomous vehicles to misinterpret stop signs vs. speed limit.
  • Bypassing facial recognition, such as the ones for ATM.
  • Bypassing spam filters.
  • Fooling sentiment analysis of movie reviews, hotels, etc.
  • Bypassing anomaly detection engines.
  • Faking voice commands.
  • Misclassifying machine learning based-medical predictions.

What adversarial effect could disrupt the world? How we can prevent it?

Improve this question
$\endgroup$
5
  • 7
    $\begingroup$ Consider that human intelligence is vulnerable to hacking $\endgroup$
    – Gaius
    Commented Jun 20, 2018 at 12:10
  • $\begingroup$ Interesting. Are you interested in "adversarial settings risk models" or something closer to a traditional cyber-security answer but still squarely about A.I.? Best wishes. $\endgroup$
    – Donate to the Edhi Foundation
    Commented Oct 13, 2019 at 1:12
  • $\begingroup$ It's not clear what you're really asking here, to be honest. You're asking too many questions or none, and that's why I've read below many different answers. This type of post is not good for Stack Exchange. We prefer more specific questions. What is your main specific question here? You're listing many possible ways to attack AI systems and then you ask "Are AI systems vulnerable to hacking?". This doesn't make any sense. What makes even less sense, because not really the same question, is the question "What adversarial effect could disrupt the world?". $\endgroup$
    – nbro
    Commented Feb 18, 2021 at 19:58
  • $\begingroup$ This post is "interesting", but I closed it because it's unclear what type of answer you're looking for and it seems that you're asking many questions. There seem to be 3 distinct questions here. 1. The one in the title "Is artificial intelligence vulnerable to hacking?", 2. "What adversarial effect could disrupt the world?" and 3. "How we can prevent it?". It seems that you're proposing ways in which AI systems could be hacked, so your question in the title is unclear. Are you looking for a "yes" or "no" answer? $\endgroup$
    – nbro
    Commented Jan 14, 2022 at 12:59
  • $\begingroup$ Do you want us to confirm that AI systems can be hacked in the way you suggest? Again, yes or no answer? The 2nd question is even more problematic. What do you mean by "disrupt the world" and "adversarial effect"? Maybe you meant "adversarial attack" rather than "adversarial effect"? Even assuming that's the case, it's not clear what you mean by "disrupt the world". It could mean many things. The 3rd question may be ok, if you clarify the second question. I would suggest that you edit your post to clarify what your question really is. $\endgroup$
    – nbro
    Commented Jan 14, 2022 at 13:03

7 Answers 7

Reset to default
21
$\begingroup$

AI is vulnerable from two security perspectives the way I see it:

  1. The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data.

  2. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with.

The first has to be mitigated in the same way as any other software. I'm uncertain if AI is any more vulnerable on this front than other software, I'd be inclined to think that the complexity maybe slightly heightens the risk.

The second is probably best mitigated by both the careful refinement of the system as noted in some of the other answers, but also by making the system more context-sensitive; many adversarial techniques rely on the input being assessed in a vacuum.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ The split between code vulnerabilities and usage vulnerabilities is good. However, the code vulnerabilities typically are minuscule in AI. The complexity of AI lies in the data, whether that's node weights in a neural network or trees in a random forest. There's just a small bit of code to feed the AI, and the chief risk there is not overfeeding it - a classic buffer overflow risk, easily mitigated by late 20th century techniques. $\endgroup$
    – MSalters
    Commented Jun 20, 2018 at 9:10
  • $\begingroup$ @MSalters I think it is hard to draw a general conclusion because the code complexity can vary a lot between different types of AI agents (I think your comment is largely accurate for neural networks). Furthermore, although the data and manipulation thereof is probably the larger attack surface it would be unwise to discount the same sort of attacks that have allowed remote code execution via compromised image files in the past that exploited flaws in image viewing applications. The vector is the data being passed in, but the behavior still falls under the code vulnerability header, I think. $\endgroup$
    – Christopher Griffith
    Commented Jun 22, 2018 at 16:47
7
$\begingroup$

Programmer vs Programmer

It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security.

eg There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ways to circumvent those steps as well. It's a pretty boring fight.

Agent vs Agent

An interesting point that @DukeZhou raised is the evolution of this war, involving two artificial intelligence (agents). In that case, the battle is one of the most knowledgeable. Which is the best-trained model, you know?

However, to achieve perfection in the issue of vulnerability, artificial intelligence or artificial super intelligence surpass the ability to circumvent the human. It is as if the knowledge of all hacks to this day already existed in the mind of this agent and he began to develop new ways of circumventing his own system and developing protection. Complex, right?

I believe it's hard to have an AI who thinks: "Will the human going to use a photo instead of putting his face to be identified?"

How we can prevent it

Always having a human supervising the machine, and yet it will not be 100% effective. This disregarding the possibility that an agent can improve his own model alone.

Conclusion

So I think the scenario works this way: a programmer tries to circumvent the validations of an AI and the IA developer acquiring knowledge through logs and tests tries to build a smarter and safer model trying to reduce the chances of failure.

Improve this answer
$\endgroup$
0
6
$\begingroup$

How we can prevent it?

There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it.

Such verifiers are done by:

This approach may help to check robustness properties of neural networks. The next step is to construct such a neural network, that has required robustness. Some of above papers contain also methods of how to do that.

There are different techniques to improve the robustness of neural networks:

At least the last one can provably make NN more robust. More literature can be found here.

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ This sounds like an impossible claim... unless it's about some particular inputs X, rather than general inputs X? In which case, it seems to say next to nothing about hackability, since inputs need not be limited to perturbations of those in the training? $\endgroup$
    – user541686
    Commented Jun 19, 2018 at 21:54
  • 1
    $\begingroup$ @Mehrdad: It's probably achievable in a probabilistic sense if the input space is sufficiently structured that you can randomly sample it. That is to say, you can probably establish that for 95% of possible inputs, 95% of disturbances smaller than ε do not affect the class label. This is equivalent to establishing that the border between output classes in input space is smooth, or that the largest part of the input space does not lie near a class border. Obviously some part of the input space has to lie near a class border. $\endgroup$
    – MSalters
    Commented Jun 20, 2018 at 9:15
  • $\begingroup$ I'm not sure this would apply in the "adversarial" case described in the paper: There, (IIRC) a back-propagated gradient is added to the whole picture, so the change to the complete input can be quite large - even if the change for each individual pixel is barely noticeable. $\endgroup$
    – Niki
    Commented Jun 20, 2018 at 10:35
  • $\begingroup$ @MSalters: I guess, yeah. But then that seems to devalue it a fair bit unless you can actually show the pictures that are on a class border should actually be on a class border... $\endgroup$
    – user541686
    Commented Jun 20, 2018 at 16:17
  • $\begingroup$ The sentence "The next step is to construct such a neural network, that has required robustness" is under research. In general it's very hard to get rid of NN non-robustness problem. But it's possible to enhance the robustness by adversarial training (see e.g. A. Kurakin et al., ICLR 2017), defensive distillation (see e.g. N. Papernot et al., SSP 2016), MMSTV defence (Maudry et al., ICLR 2018). At least the last one can provably make NN more robust. $\endgroup$
    – Ilya Palachev
    Commented Jun 21, 2018 at 17:29
4
$\begingroup$

I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision).

https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google

https://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms

Improve this answer
$\endgroup$
4
$\begingroup$

Is Artificial Intelligence Vulnerable to Hacking?

Invert your question for a moment and think:

What would make AI at less of a risk of hacking compared to any other kind of software?

At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI doesn't grant it some kind of immunity.

As for AI-specific tampering, AI is at risk to being fed false information. Unlike most programs, AI's functionality is determined by the data it consumes.

For a real world example, a few years ago Microsoft created an AI chatbot called Tay. It took the people of Twitter less than 24 hours to teach it to say "We're going to build a wall, and mexico is going to pay for it":

We're going to build a wall, and mexico is going to pay for it

(Image taken from the Verge article linked below, I claim no credit for it.)

And that's just the tip of the iceberg.

Some articles about Tay:

Now imagine that wasn't a chat bot, imagine that was an important piece of AI from a future where AI are in charge of things like not killing the occupants of a car (i.e. a self-driving car) or not killing a patient on the operating table (i.e. some kind of medical assistance equipment).

Granted, one would hope such AIs would be better secured against such threats, but supposing someone did find a way to feed such an AI masses of false information without being noticed (after all, the best hackers leave no trace), that genuinely could mean the difference between life and death.

Using the example of a self-driving car, imagine if false data could make the car think it needed to do an emergency stop when on a motorway. One of the applications for medical AI is life-or-death decisions in the ER, imagine if a hacker could tip the scales in favour of the wrong decision.

How we can prevent it?

Ultimately the scale of the risk depends on how reliant humans become on AI. For example, if humans took the judgement of an AI and never questioned it, they'd be opening themselves up to all sorts of manipulation. However, if they use the AI's analysis as just one part of the puzzle, it would become easier to spot when an AI is wrong, be it through accidental or malicious means.

In the case of a medical decision maker, don't just believe the AI, carry out physical tests and get some human opinions too. If two doctors disagree with the AI, throw out the AI's diagnosis.

In the case of a car, one possibility is to have several redundant systems that must essentially 'vote' about what to do. If a car had multiple AIs on separate systems that must vote about which action to take, a hacker would have to take out more than just one AI to get control or cause a stalemate. Importantly, if the AIs ran on different systems, the same exploitation used on one couldn't be done on another, further increasing the hacker's workload.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I like the idea of having several separate AI systems that have to reach agreement as a mitigation technique. Although then you'd have to be assured whatever voting mechanism they used couldn't be comprised to fake a decision. $\endgroup$
    – Christopher Griffith
    Commented Jun 22, 2018 at 17:37
  • $\begingroup$ @ChristopherGriffith True, that is a risk. In the case of the car, the best way to mitigate that is to design the system so that an attacker would need physical access to manipulate it and make it hard to reach so the person would have to break in to the car to access it. Keeping a system offline is generally a good hacking countermeasure, though not always ideal. $\endgroup$
    – Pharap
    Commented Jun 22, 2018 at 22:26
1
$\begingroup$

I concur with Akio that no system is completely safe, but the take away is AI systems are less prone to attacks when comparing with the old systems because of the ability to constantly improve.

As time passes by more people will get in the field bringing new ideas and hardware will be improving so that they are "strong AI."

Improve this answer
$\endgroup$
0
$\begingroup$

There are many ways to hack an AI. When I was kid I figured how to beat a chess computer. I always followed the same pattern, once you learn you can exploit it. The worlds best hacker is a 4 year old that wants something he will try different things until he establishes pattern in his parents. Anyway, Get an Ai to learn the patterns of a AI and given a given combination you can figure the outcome. There is also just plain flaws or back door in code either on purpose or by chance. There is also the possibility the AI will hack itself. It is called misbehaving, remember the small child again...

BTW simple way is to make AI always fails safe... something people forget.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged .