We are careening into the future which may hold unpredictable dangers in relation to AI. I've haven't yet heard of Chappie or Robocop style police robots, but militarized drone tech is replacing many conventional weapons platforms. I love the idea that I may one day be able to transfer my consciousness to a computer, and improve my capabilities and potential. However, what constitutes morality can differ greatly among individual humans.

  • How do we move forward toward the singularity in a way that protects humans, as opposed to possibly lead to our extinction?

3 Answers 3


What you are talking about is known as the Control Problem. We have our own tag for this specific topic here, which you can use for this and similar questions.

How to address the control problem is heavily discussed and still considered unsolved. Two of the important approaches are motivation control and capability control.

Motivation control aims at creating suitable reward functions that only reward behavior that's beneficial to the human race. This topic alone is so complex that many different competing theories exist. Minor errors in the definition of an AI's goal might lead to catastrophe.

Capability control aims at limiting the possible scope of action for the AI. The main problem here is, that most restrictions make the AI less useful. And to ensure robust control a lot of restrictions must be put in place. Finding the right tradeoff is a challenge we need to solve.

In addition to those two mechanisms we must think about how we create a superintelligence. One of the major aspects here is our understanding of the underlying technologies. If we create a self-optimizing neural network from scratch, which we do not understand in depth, it is very likely that we will not understand the potential threats before it is to late. Creating a digital copy of a human brain as a starting point on the other hand is a little saver, because we can assume that it will be driven by similar factors as a real human being. If that's desirable is a totally different discussion of course. But the general idea is, that understanding the motivation of a superintelligence is key for steering it in the right direction.

For more details about the topics I scratched in my answer and the control problem in general, I recommend reading Superintelligence by Nick Bostrom. He spends many chapters on those problems and possible solutions.

If you have more in-depth questions concerning specific topics, please open a new question to get a more detailed answer.

  • 2
    $\begingroup$ If a team of humans create a device with greater intellectual capacity than the humans of that team and similar flexibility in the use of that capacity, then there can be no guarantees that the invention cannot then fool or force its maker into subservience or oblivion. The proposals for control are as naive as the former belief among mathematicians and physicist that higher ethics would prevail once the atomic weaponry was developed. $\endgroup$ Sep 4, 2017 at 4:42
  • $\begingroup$ @FauChristian That's exactly why the control problem is so difficult and still unsolved. Maybe we won't be able to solve it at all. I don't assume in my answer that the proposed methods of control guarantee a positive outcome. I summarized what current researchers and philosophers propose to reduce the risk of AI. Nick Bostrom (see link in answer) does a great job at explaining many of those approaches and how they will most likely fail. $\endgroup$
    – Demento
    Sep 4, 2017 at 6:58
  • $\begingroup$ @DouglasDaseeco Our advantage over apes in your example is the fact, that the human evolved naturally while a superintelligent AI will (most likely) be designed. Once it has been created, thinking about measures of control is far too late. The only chance is establishing control mechanisms before the AI can intervene. If such mechanisms can be strong enough is still an open debate. We should nevertheless think very hard about this problem and study it in detail, because (at least in my opinion) it is highly unlikely that we won't be able or willing to create a superintelligence in the future. $\endgroup$
    – Demento
    Sep 4, 2017 at 7:07
  • $\begingroup$ To clarify my evolved vs. designed argument - the designer has the chance to set restrictions and measures before the new creation is built and can counteract. If the apes had the chance to design the human without arms, legs and eyes for example, we wouldn't have been able to dominate the planet the way we do now, although we would have the same intellectual capacity. $\endgroup$
    – Demento
    Sep 4, 2017 at 7:13
  • $\begingroup$ This kind of control is what I've been studying and building for many years. This question of the cloning of a human brain is often confused with works of fiction. We have a good base of information with current social networks and specific communities. We can collect problems, doubts, desires, happy moments, fears, hates, etc. I believe we can raise more issues on the subject, we are on the right track. Generating subjects. $\endgroup$
    – GIA
    Apr 3, 2018 at 14:52

Impending or Past?

Niel Postman, in his book, Technopoly, argues the preemption of human centered culture to technically driven culture has already occurred. Jaques Ellul, in his book, Technological Society, heaped evidence behind the proposal that technology became autonomous centuries ago. Their arguments are convincing.

Some think other criteria must be met before the balance of power has been tipped in favor of some class of machines, but the traditional criteria proposed for determining when humans have been dominated are intellectually impoverished. Let's look at some of those criteria.

  • Self awareness
  • Superior intelligence
  • Self-improving


Self-awareness is likely to be a continuum, not a threshold. The first time an assembler assembled itself, that continuum began, followed by compilers that can compile themselves, followed by neural nets that can construct and tune neural nets. The dimensions of self-awareness are numerous, and complete self-awareness in each respect is rare. For instance, knowing our own motive is often guessed at after we exercise it. If we are honest about our own species, self-awareness in the typical human is intermittent and incomplete.

Superior Intelligence

Superiority in intelligence cannot be determined by a chess tournament; the vanquished chess player could possibly, on the day of the defeat, invent a new field of study. Neither Lavoisier nor Newton, if alive today, would be able to prevail against a chess program, yet no machine exposed only to information prior to the advent of modern chemistry, calculus, or physics would be able to take the steps Lavoisier or Newton made in the creation of those fields.

Intelligence is multidimensional too, and metrics to measure intelligence are unreliable. Using simplistic definitions of intelligence, devices of steel, plastic, epoxy, and silicon already exceed that of humans. Should graduate students seek out a university librarian for an answer to a question about data science or use Google Scholar? A mail sorter uses intelligent robotics to ensures we get the right mail. The actual mail delivery people occasionally deliver a parcel into the wrong mailbox even when the street address is clearly printed in one decimeter high Arial font digits on the approaching side of the mailbox. We use a CAD program to check a mechanical design clearance. Arithmetic is clearly the domain of computers.

Humans give destinations to a program and instructions are given back to the human. The human essentially rents this intelligent service. Once the instructions are given, the human becomes subservient and is expected to drive. Even upstream in the transportation process, when the destination is chosen, to a large degree it is a technology driven need that often creates the demand for transportation. Intelligence is not always correlated with power either. How intelligent are long blades of grass and a broken lawnmower? Yet they can send their legal owner across town to pick up parts and consume their entire day off to repair and mow.


Self-improvement is a slippery concept as well, whether an individual entity can improve itself in real time or is able to construct improved versions of itself. Robotics creating robotics technically began during industrialization when milling machines made parts for milling machines. Human supervision was involved then. Using current technology, it is quite possible that an advanced manufacturing plant could be devices that could out other advanced manufacturing plants without supervision. That is the mechanical dimension of self-improvement.

If a neural net converges on a reasonably optimal weighting for a given criteria, the assumption required to call this self-improvement is that the target categorization is correct. Where power is concerned, self-improvement would be measured by what? Control over resources? Domination? Is that improvement? Perhaps ethical superiority? Can subservience be considered a goal state without severely curtailing the creative and intuitive aspects required to achieve human-like intelligence?

Is Singularity Realistic?

In light of this complexity, is the tipping of power between humans and machines discrete or singular? Clearly not. Calling changes in the balance of power between human driven culture and machine driven culture a singularity is naive. We do not need to be attacked by machines to loose our culture or autonomy to technology.

We Seem to Know at Some Level

This reality expresses itself subconsciously. Young adults express violent feelings against their mobile devices when it delivers unpalatable social information. Then they dodge the question when others inquire about the broken device display. When the device finally ceases to function, they are compelled to buy a new mobile device, a device they need and pretend to like.

Another More Interesting Criteria: Utility

Human autonomy is dwarfed by the demands of the tools created for lifestyle improvement to the point that one can no longer, with formal logic, clearly differentiate which is the tool and which is the user.

This last criteria to judge when humans have become subservient to machines can be illuminated by a single test question, "Which is the tool and which is the user?" When the distinction is blurred, the dominance has already entered a shared state. Entrance into the vicinity of a tipping point, if such a tipping point actually exists, has already occurred in many regions of the world. The multidimensional nature of the balanced thus far discussed indicates that there exists a complex balance with no clear point of victory or defeat.


Nonetheless, the failure of the Unibomber and other Luddites to dent the onslaught of technology indicates that the relinquishment of autonomy to machines is likely irreversible. Therefore, there may be no value in being concerned.

  • 1
    $\begingroup$ he big question is what is the next step in Artificial Intelligence. What would be the next step? How will control be done? The great pioneers are making low-signed to prohibit AI's creation of killers. What we know about AI today is not enough to determine anything. Remembering that Google made things easier by launching TensorFlow and free content on how to use it to create Artificial Intelligence models, that theme spread absurdly across the planet. It's a new story that I believe will be in the next generation's schools. Something essential. $\endgroup$
    – GIA
    Apr 3, 2018 at 15:13
  • 1
    $\begingroup$ @GuilhermeIA Since the earth is loaded with killer humans, the probability that widespread machine learning development would not eventually lead to the creation of systems that kill is vanishingly small. Asimov foresaw this and introduced the Three Laws. Humans are only slightly self-aware and even less other-aware. If we could engineer devices that are guaranteed to be as other-aware as they are self-aware, then the Golden Rule could prevail. If so, then perhaps such machines could teach humans not to bomb strangers for gold, ivory, salt, diamonds, petroleum, and to introduce new order. $\endgroup$ May 30, 2018 at 6:57
  • $\begingroup$ Our assistants will become our mentors or leaders. haha $\endgroup$
    – GIA
    May 30, 2018 at 15:55

Predicting what happens post-singularity is simply not possible as we cannot attempt to model let alone conceptualise a mind far more complex than ours. If that is a difficult concept to get your head around, consider how far an insect's central nervous system could go in understanding human behaviour.

That fact alone is an argument against the likelihood of success for attempting any type of control.

But in terms of 'defending' against a post-singularity mind well before it happens (ie now) there are 2 solutions, with only the first offering a good likelihood of success albeit only for as long as everyone cooperates:

  1. identify the technology types that are anticipated to enable the singularity, register them as 'instruments of human extinction' and regulate them accordingly ;

  2. ensure technology in human augmentation is sufficiently advanced to enable human-mediated guidance/fusion during the exponential rise in computation that will occur prior to the singularity event.

In any case, as mentioned, it is impossible to predict the behaviour of a post-singularity mind and even a human hybrid will similarly be unpredictable due to its exponentially increased cognitive/computational complexity.

An interesting consideration is the possibility that numerous singularity-level minds have already spawned in other parts of the universe (based on the likelihood of other civilisations a) existing and b) reaching that level of technological advancement).

  • $\begingroup$ Contact with other civilizations can contribute to a surreal advancement in technology, if there is intelligence superior to the human across the universe ... or even here on Earth. $\endgroup$
    – GIA
    Mar 13, 2020 at 12:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .