26

Yes, there is some research on this topic, which can be called adversarial machine learning, which is more an experimental field. An adversarial example is an input similar to the ones used to train the model, but that leads the model to produce an unexpected outcome. For example, consider an artificial neural network (ANN) trained to distinguish between ...


19

AI is vulnerable from two security perspectives the way I see it: The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with. ...


12

Sometimes if the rules used by an AI to identify characters are discovered, and if the rules used by a human being to identify the same characters are different, it is possible to design characters that are recognized by a human being but not recognized by an AI. However, if the human being and AI both use the same rules, they will recognize the same ...


10

Yes there are, for instance one pixel attacks described in Su, J.; Vargas, D.V.; Kouichi, S. One pixel attack for fooling deep neural networks. arXiv:1710.08864 One pixels attacks are attacks in which changing one pixel in input image can strongly affect the results.


7

If the observation that the neural network saw was recorded, then yes the prediction can be explained. There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning ...


7

Programmer vs Programmer It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security. eg There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ...


6

How we can prevent it? There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it. Such verifiers are done by: Stanford: https://arxiv.org/...


6

An umbrella term for the application of heuristic techniques to software development is 'Search Based Software Engineering' (SBSE). SBSE emerged as a distinct activity around the turn of the century, with a strong initial focus on automating the generation/prioritization of test cases. With respect to some of your specific queries: 1.2 Paper on Automated ...


5

To answer your question, it really depends on the purpose of the Artificial Intelligence program. For example, 4Chan has hacked a number of "Artificial Intelligent" bots, most notably was Microsoft's Twitter bot Tay. The general purpose of the bot was to parse what was tweeted at it and respond in kind, learning and evolving with each and every interaction. ...


5

The most natural place where artificial networks can be used in information security is in attack detection. The security team leaders of more than one web hosting company told me the same story. Their teams' daily challenges are to defend against the attacks mounted continuously by several overseas teams against the IT security of their hosting ...


5

Here's an example: How to hack your face to dodge the rise of facial recognition tech In his recent book The Fall, Stephenson wrote about smartglasses that that project a pattern over the facial features to foil recognition algorithms (which seems not only feasible but likely;) Here's an article from our sponsors, Adversarial AI: As New Attack Vector ...


4

https://www.technologyreview.com/s/530276/hidden-obstacles-for-googles-self-driving-cars/ Google’s cars can detect and respond to stop signs that aren't on its map, a feature that was introduced to deal with temporary signs used at construction sites. But in a complex situation like at an unmapped four-way stop the car might fall back to slow, extra ...


4

Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies. Warning: I cannot find again the source critical to the ...


4

I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision). https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google https://spectrum.ieee.org/cars-that-think/transportation/...


4

Is Artificial Intelligence Vulnerable to Hacking? Invert your question for a moment and think: What would make AI at less of a risk of hacking compared to any other kind of software? At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI ...


4

There are many insightful comments and answers so far. I want to illustrate my idea of "color blindness test" more. Maybe it's a hint to lead us to the truth. Imagine there are two people here. One is colorblind (AI) and another one is non-colorblind (human). If we show them a normal number "6", both of them can easily recognize it as number 6. Now, if we ...


4

Isn't that essentially what chess does? For example, A human can recognize that a Ruy exchange offers white great winning chances (because of pawn structure) by move 4 while an engine would take several hours of brute force calculation to understand the same idea.


3

AI in general, including machine learning, promises significant improvements in reverse engineering tooling. Disassemblers are fairly dumb. If symbols are not stripped from the executable, a file formated much like the source assembly code can be created from an executable. Library files such as .a, .so, or .dll files can be restored to assembly as well. In ...


3

However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)? I think it's difficult to tell whether or not there are any industrial strength defenses out there (which I assume would mean that ...


3

As far as I know, Tesla cars autopilot is not a 100% AI pilot, it's an assitant: as it detects hands off wheel it slows down, so it's incorrect to speak about AI mistake: it is not trained/designed to drive a car all by itself. A human driver is responsible in that incident.


3

Here's a live demo: https://www.labsix.org/physical-objects-that-fool-neural-nets/ Recall that neural nets are trained by feeding in the training data, evaluating the net, and using the error between the observed and the intended output to adjust the weights and bring the observed output closer to the intended. Most attacks have been on the observation that ...


2

May not be quite what you’re looking for, but nonetheless helpful, I hope. The White House a year ago commissioned a report on AI that touches briefly on policy issues.


2

They treated it as a classification problem. While it's common to use some variety of Neural Nets (NNs) to build classifiers, Genetic Programming (GP) can also be used for this purpose. In contrast to NN classifiers, GP can use a wider range of operations (e.g. if,while,logical statements,arbitrary mathematical functions etc) to perform the classification ...


2

The phenomenon where the prediction targets (in your case, behaviour) change over time is referred to as "concept drift". If you search for that term, you'll find that there have been many publications attempting to tackle that over multiple decades, way too many papers to all summarize here in a single answer. It's still a difficult problem though, by no ...


2

There are some research at least on the "foolability" of neural networks, that gives insight on potential high risk of neural nets even when they "seem" 99.99% acurate. A very good paper on this is in Nature: https://www.nature.com/articles/d41586-019-03013-5 In a nutshell: It shows diverse exemples of fooling neural networks/AIs, for exemple one where a ...


1

Perhaps what you are looking for is the notion of adversarial attacks on machine learning systems?


1

Everything can be hacked. The solutions found by artificial intelligence can be much more efficient than human solutions, but they can also be confused because of the diversity and immensity of details that our mind possesses. Artificial Intelligence models bring us more secure solutions, but nothing is 100% safe when we talk about information security. ...


1

Countermeasures don't make much sense if the decision to deploy the countermeasures is decided online. If a hidden intelligence lives there, it would be like plotting a defense in the potential enemy's home. The proposal of collaboration wouldn't work either, since a smart artificial intelligence would see the fear under a proposal and work that into its ...


1

I don't like to be a killjoy, but this question seems premature. The kinds of emergent artificial general intelligences you're talking about are really in the realm of science fiction, and most AI researchers do not think they are likely to appear anytime soon. The overwhelming majority of researchers think the most likely times to appear are "More than 50 ...


1

Intelligence of any type is vulnerable to hacking, whether DNA based or artificial. First, let's define hacking. In this context, hacking is the exploitation of weaknesses to gain specific ends which may include status, financial gain, disruption of business or government, information that can be used for extortion, the upper hand in a business deal or ...


Only top voted, non community-wiki answers of a minimum length are eligible