28

Yes, there is some research on this topic, which can be called adversarial machine learning, which is more an experimental field. An adversarial example is an input similar to the ones used to train the model, but that leads the model to produce an unexpected outcome. For example, consider an artificial neural network (ANN) trained to distinguish between ...


21

AI is vulnerable from two security perspectives the way I see it: The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with. ...


12

Sometimes if the rules used by an AI to identify characters are discovered, and if the rules used by a human being to identify the same characters are different, it is possible to design characters that are recognized by a human being but not recognized by an AI. However, if the human being and AI both use the same rules, they will recognize the same ...


11

Yes there are, for instance one pixel attacks described in Su, J.; Vargas, D.V.; Kouichi, S. One pixel attack for fooling deep neural networks. arXiv:1710.08864 One pixels attacks are attacks in which changing one pixel in input image can strongly affect the results.


7

An umbrella term for the application of heuristic techniques to software development is 'Search Based Software Engineering' (SBSE). SBSE emerged as a distinct activity around the turn of the century, with a strong initial focus on automating the generation/prioritization of test cases. With respect to some of your specific queries: 1.2 Paper on Automated ...


7

If the observation that the neural network saw was recorded, then yes the prediction can be explained. There was a paper written fairly recently on this topic called "Why Should I Trust You?": Explaining the Predictions of Any Classifier (2016). In this paper, the author described an algorithm called LIME which is able to explain any machine learning ...


7

Programmer vs Programmer It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security. eg There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ...


6

How we can prevent it? There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it. Such verifiers are done by: Stanford: https://arxiv.org/...


5

To answer your question, it really depends on the purpose of the Artificial Intelligence program. For example, 4Chan has hacked a number of "Artificial Intelligent" bots, most notably was Microsoft's Twitter bot Tay. The general purpose of the bot was to parse what was tweeted at it and respond in kind, learning and evolving with each and every interaction. ...


5

Here's an example: How to hack your face to dodge the rise of facial recognition tech In his recent book The Fall, Stephenson wrote about smartglasses that that project a pattern over the facial features to foil recognition algorithms (which seems not only feasible but likely;) Here's an article from our sponsors, Adversarial AI: As New Attack Vector ...


5

https://www.technologyreview.com/s/530276/hidden-obstacles-for-googles-self-driving-cars/ Google’s cars can detect and respond to stop signs that aren't on its map, a feature that was introduced to deal with temporary signs used at construction sites. But in a complex situation like at an unmapped four-way stop the car might fall back to slow, extra ...


5

The most natural place where artificial networks can be used in information security is in attack detection. The security team leaders of more than one web hosting company told me the same story. Their teams' daily challenges are to defend against the attacks mounted continuously by several overseas teams against the IT security of their hosting ...


4

However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)? I think it's difficult to tell whether or not there are any industrial strength defenses out there (which I assume would mean that ...


4

Is Artificial Intelligence Vulnerable to Hacking? Invert your question for a moment and think: What would make AI at less of a risk of hacking compared to any other kind of software? At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI ...


4

I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision). https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google https://spectrum.ieee.org/cars-that-think/transportation/...


4

Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies. Warning: I cannot find again the source critical to the ...


4

There are many insightful comments and answers so far. I want to illustrate my idea of "color blindness test" more. Maybe it's a hint to lead us to the truth. Imagine there are two people here. One is colorblind (AI) and another one is non-colorblind (human). If we show them a normal number "6", both of them can easily recognize it as number 6. Now, if we ...


4

Isn't that essentially what chess does? For example, A human can recognize that a Ruy exchange offers white great winning chances (because of pawn structure) by move 4 while an engine would take several hours of brute force calculation to understand the same idea.


3

Here's a live demo: https://www.labsix.org/physical-objects-that-fool-neural-nets/ Recall that neural nets are trained by feeding in the training data, evaluating the net, and using the error between the observed and the intended output to adjust the weights and bring the observed output closer to the intended. Most attacks have been on the observation that ...


3

As far as I know, Tesla cars autopilot is not a 100% AI pilot, it's an assitant: as it detects hands off wheel it slows down, so it's incorrect to speak about AI mistake: it is not trained/designed to drive a car all by itself. A human driver is responsible in that incident.


2

May not be quite what you’re looking for, but nonetheless helpful, I hope. The White House a year ago commissioned a report on AI that touches briefly on policy issues.


2

They treated it as a classification problem. While it's common to use some variety of Neural Nets (NNs) to build classifiers, Genetic Programming (GP) can also be used for this purpose. In contrast to NN classifiers, GP can use a wider range of operations (e.g. if,while,logical statements,arbitrary mathematical functions etc) to perform the classification ...


2

There are some research at least on the "foolability" of neural networks, that gives insight on potential high risk of neural nets even when they "seem" 99.99% acurate. A very good paper on this is in Nature: https://www.nature.com/articles/d41586-019-03013-5 In a nutshell: It shows diverse exemples of fooling neural networks/AIs, for exemple one where a ...


2

The phenomenon where the prediction targets (in your case, behaviour) change over time is referred to as "concept drift". If you search for that term, you'll find that there have been many publications attempting to tackle that over multiple decades, way too many papers to all summarize here in a single answer. It's still a difficult problem though, by no ...


2

It will recover the encrypted inputs. The algorithm starts with dummy data and dummy labels, and then iteratively optimizes the dummy gradients to be close as to the original. This makes the dummy data close to the real training data: $$\mathbf{x}^{\prime *}, \mathbf{y}^{\prime *}=\underset{\mathbf{x}^{\prime}, \mathbf{y}^{\prime}}{\arg \min }\left\|\nabla W^...


2

You have implicitly assumed that supervised learning is being used, given the assumption that labels are needed. But this might lead to the following potential problems: Log file data tends to be huge, and it may be infeasible to label due to the time/expertise required; Then there's the class imbalance problem, in that attack examples are far far rarer ...


1

While @Oliver Mason's comment is correct, and your proposed method won't provide perfect security, you can still protect your models at rest, so that they are stored encrypted in the memory, and your software feed the key at runtime to decrypt it. On whatever DL inference engine that you have, once it supports loading the model from a buffer (e.g. void*) ...


1

Perhaps what you are looking for is the notion of adversarial attacks on machine learning systems?


1

Everything can be hacked. The solutions found by artificial intelligence can be much more efficient than human solutions, but they can also be confused because of the diversity and immensity of details that our mind possesses. Artificial Intelligence models bring us more secure solutions, but nothing is 100% safe when we talk about information security. ...


1

Another point of view - In safety-critical real world systems, this attack should be evaluated from other aspects as well. In many systems the attack is somewhat mitigated to physical attacks only - for example, you can't add digital noise to a camera used for autonomous driving - you need to print an adversarial e.g. stop sign and locate it in a place, ...


Only top voted, non community-wiki answers of a minimum length are eligible