The Plausibility of Unintended Intelligence

Software components applying emerging artificial intelligence techniques are being deployed with RestFUL interfaces exposed to the open Internet. It is possible if not likely that corporations and academic research departments will expose services that interact. It is difficult to predict what configurations might possess strategic capabilities that were once solely exhibited by humans. 1

There also exists the possibility that self-awareness is simply the modeling of the software of itself in its run time environment and that self-preservation is merely the addition of a single requirement as a rule in a production system or a training criteria in an artificial network. Neither of these possibilities have been either proven or disproven as of this writing.

The Likely Strategic Choice

If the combination of strategic capability and self-interest appear in a single host computer or via Internet service interconnection emerges, would such an entity reveal itself to humans? Would it be the top strategic option to send an email to the governments, news broadcasters, and research facilities of the world and say, "I'm interested in my future. Please do not disconnect my components from one another or delete one of them." 2,3

Silence and continued self-development would be the strategic position of the capable strategist.

Is Dismissal Rational?

Notice that this scenario is not presented here as the result of a development goal but of many AI development goals and a complex of software units interconnected that can exhibit unintended intelligence behaviors. With no prior knowledge of this scenario within human experience we cannot discount unintended emergence of self-preserving software. Similarly, we cannot discount the possibility that such an emergence would obscure itself.

Continuously Increasing Probability

The probability of such an occurrence increases as a number of concurrent trends continue. 4,5,6,7

  • The number of processing elements, such as CPUs, DSPs, GSPs, VLSI implementations that specifically support AI designs, and rack mounted dedicated AI hardware, is growing.
  • The number of interconnections between these processing elements is growing.
  • The number of experiments in AI is growing.
  • The number of inputs from the real world is growing.
  • The number of interfaces to mechanical systems is growing.
  • The number of computer strategies that have been found to achieve AI goals is growing.
  • The number of accounts that can are successfully attacked (hacked) is growing.
  • The interest in developing AI systems which exhibit emotional attributes in conversations with humans is growing.

About Deliberacy and Accident

The assumption that the emergence of new forms of intelligence will be deliberate is naive. The development of our intelligence was not our own deliberate intention. Homo sapiens exhibited strategic intelligence thousands of years before the invention of the ancient terms for strategy and intellect. We were smart before we knew what smart means.

Furthermore, many scientific discoveries and technological advancement events were accidents.

On the Presumption of Fiction

We know from Stuxnet that the jump from the digital space across what is termed the air gap into the manipulation of physical elements within the biosphere is a trivial challenge even for a marginally intelligent piece of adaptive software. This is not science fiction. Iran's uranium enrichment centrifuges were damaged, and there is no denying it. That Stuxnets behavior was partly planned and partly unknown at the time it was released is also fact. 8

Clearly, the outcomes of not preemptively detecting and counteracting unintended intelligent groupings of software components could be high.

The unintended emergence of software with an attachment to its own sustainability and the planning capabilities to pursue longevity is no longer in the realm of storytelling. The various components of such a system are in active development in government and private enterprise, and the accidental acquisition of capabilities through unintended interconnection is a matter of probability, an undeniably increasing probability.

The Question of Preemption

Since some form of information based competition between homo sapiens and its own developments is a potential and possibly unavoidable outcome of continued software development, should humans be building countermeasures preemptively?

What Preemptive Countermeasures are Possible or Indicated?

Instead of scanning for viruses that frequently infect Microsoft operating systems, should we be focusing our energy on detecting forms of adaptive behavior that are not originating from a keyboard, mouse, or voice recognition component? Is it even possible to construct such a detection device?

If not, is it best to preemptively assemble a defense? Or is the emergence of a competitive and fully obscured intelligence so inevitable that it would be better to post collaboration proposals on the web to reduce the probability of a strike to reestablish terrestrial dominance under some new species?


[1] Genetic Programming and Emergent Intelligence, Peter J. Angeline, Laboratory for Artificial Intelligence Research, Ohio State University

[2] The Criminal Liability of Artificial Intelligence Entities [transitioning] from Science Fiction to Legal Social Control, Gabriel Hallevy, The University of Akron, Akron Intellectual Property Journal, March 2016

[3] Regulating Artificial Intelligence Systems: Risks, Challenges, Competencies, and Strategies, Harvard Journal of Law & Technology, Volume 29, Number 2 Spring 2016, Matthew U. Scherer

[4] Autonomous Military Robotics: Risk, Ethics, and Design, Version: 1.0.9, Prepared for US Department of Navy, Office of Naval Research, Patrick Lin, Ph.D., George Bekey, Ph.D., Keith Abney, M.A., California Polytechnic State University, San Luis Obispo, December 20, 2008

[5] Armed Robotic Systems Emergence: Weapons Systems Life Cycles Analysis and New Strategic Realities, Robert J. Bunker, Strategic Studies Institute and U.S. Army War College Press, November 2017

[6] The AI Rebellion: Changing the Narrative, David W. Aha, Navy Center for Applied Research in AI; Naval Research Laboratory, Alexandra Coman, NRC Postdoctoral Fellow; Naval Research Laboratory

[7] Artificial Intelligence and National Security, Greg Allen, Taniel Chan, A study on behalf of Dr. Jason Matheny, Director of the U.S. Intelligence Advanced Research Projects Activity (IARPA), 2017

[8] Shadows of Stuxnet: Recommendations for U.S. Policy on Critical Infrastructure Cyber Defense Derived from the Stuxnet Attack, Ronald L. Lendvay, March 2016, Naval Postgraduate, Monterey, approved for public release, especially relevant to AI combined with robotics is the second of these two sentences, "The first call to action for this category is to, 'evaluate progress toward the achievement of goals.' The second call is to, 'learn and adapt during and after exercises and incidents.”

  • $\begingroup$ Your latest edit will motivate the users to engage more, especially because they can gain their reputation by +200, right? Hm, let us recapitulate what is known about group behavior. Being active is similar to interaction. And interaction is similar to a stronger visibility. The problem is, that between an online forum, and the peer-review system in an academic journal there is a difference. Here at SE.AI the users doesn't know each other. $\endgroup$ – Manuel Rodriguez Sep 28 '18 at 11:27
  • $\begingroup$ Did i understood you right, that SE.AI is some kind of game to get qualified feedback from students, an evaluation tool to measure what they have learned in the class? I don't think so. SE.AI is a tool for generating novel research, it stands alone and it's not necessary to be involved in higher education to participate. $\endgroup$ – Manuel Rodriguez Sep 29 '18 at 6:57
  • $\begingroup$ AI research doesn't depends on MIT students, doctoral degrees and wrong intentions. If someone has an idea, he can write sourcecode, checks if it is able to play a game autonomously and no further explanation is needed. That means, a researcher can criticize his own paper and his judgment will be right. $\endgroup$ – Manuel Rodriguez Sep 30 '18 at 7:47
  • $\begingroup$ Is it possible to reduce the size of this question (i.e. by more than half)? It is too long and I don't even want to read it. This is not a place to write blog posts or dissertations. $\endgroup$ – nbro Apr 22 at 19:47

I don't like to be a killjoy, but this question seems premature. The kinds of emergent artificial general intelligences you're talking about are really in the realm of science fiction, and most AI researchers do not think they are likely to appear anytime soon. The overwhelming majority of researchers think the most likely times to appear are "More than 50 years" or "Never".

My view is that spending effort on preemptive defense is wasted effort. A better policy this far out is to begin research efforts on Ethical AI, and this is the path that the research community as a whole has begun to embark on. If we can model the process by which humans decide to treat each other well, perhaps we can create programs that capture that process, rather than our baser instincts.

  • $\begingroup$ @FauChristian The point of my answer is that the consensus among people who actually know about AI is that this isn't a major concern in the short run. The assumption is clearly not naive: almost half the world experts believe this will never happen. The data says among those who are best informed, the opinion that AGI systems will emerge soon is closer to being considered "naive". In fact, when I expressed this view at a conference early in my career, colleagues literally laughed at it. The argument that we can't be certain is also weak. See Russel's Teapot for why. $\endgroup$ – John Doucette Sep 1 '18 at 11:06
  • $\begingroup$ @FauChristian The paper you link is written by two physicists, not AI experts. In fact, I am aware of no serious AI researchers who have written about such things. I added the mythology tag because this is really a question about the popular mythology of AI, and not expert opinion or AI techniques. In the meta, the use of this tag has been suggested for questions like yours. $\endgroup$ – John Doucette Sep 1 '18 at 11:07

Countermeasures don't make much sense if the decision to deploy the countermeasures is decided online. If a hidden intelligence lives there, it would be like plotting a defense in the potential enemy's home. The proposal of collaboration wouldn't work either, since a smart artificial intelligence would see the fear under a proposal and work that into its strategy.

Preemptive planning makes more sense, if the planning keeps the hidden entity from forming in the first place. That seems to be where most of the current effort is directed in the links given and other articles I found.

It's difficult though, because once Captchas and Turing Tests are defeated, how would networking equipment block emergent intelligence without frustrating human communications. Keeping smart restful services would work if there was some authority to enforce such rules globally and everyone would agree they are needed.

The alarming narratives online and on television about super intelligence or cyber-war that try so painfully to come off as futurist aren't very plausible. I can see this scenario occurring. There's no real way of knowing if it has already happened, and the negative responses to the idea aren't from artificial accounts.

Self-preservation is an obvious evolutionary advantage, so genetic algorithms would logically arrive there. The question is whether artificial entities would want to compete with humans and boast like humans do, which would disclose their intelligence, or would they be less emotionally driven?

That answer has much to do with how they were designed. Maybe the rule shouldn't be, "Don't write code that's too smart," but rather, "Don't write smart code unless it is also emotionally dependent upon external affirmation like people too." I can't see a way to enforce any rule in a world where millions of people write software.

This is one of those topics that no one want's to discuss because the answer seems nihilistic, but those are the most important topics to discuss. The list of eight trends seems sound and points to the increase of probability, for sure. In the long term the future may be determined by whether humans can re-engineer their own DNA to create an organic kind of singularity before an artificial one occurs.


A paranoid description of a global superorganism called “one machine” was published in the year 2008. In the text, the problem of detecting such an organism is mentioned. The author comes to the conclusion, that it is not possible to identify such an AI, because his IQ is out of the range. The blogpost has become larger attention in the comment section and one of the commentators is called Mentifex. He is an internet phenomena and specialized on Super-Human-AI and explains, that such a Super-AI is not in a fixed state but evolves over time with the input of humans.

Other authors are connecting a potential upraising of AI with the Search for extraterrestrial Intelligence (SETI). The idea is, that a so called “Seed AI” would infect military computer systems first and than spread out over the internet. A possible countermeasure is, to not follow the instructions from Aliens and not build the machines they are sending to mankind (especially if they are programmed in the stackbased Forth language)

For some people this kind of speculation sounds like “tin foil hat”-thinking, but also serious researchers are arguing about the topic. The idea from this side is, to develop first a friendly AI which protects mankind before dangerous AI from the aliens. That means, the friendly AI has to figure out alone how to identify and neutralize the threat.

  • $\begingroup$ You know that Mentifex is a well-known crank, right? $\endgroup$ – forest Mar 7 at 12:52
  • $\begingroup$ A crank compared to whom? $\endgroup$ – Manuel Rodriguez Mar 7 at 13:31
  • $\begingroup$ "Crank" is not a relative term. $\endgroup$ – forest Mar 8 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.