If the AI goal is to serve humans and protect them (if this ever happens) and AI someday realizes that humans destroy themselves, will it try to control people for their own good, that is, will it control man's will to not destroy himself?

  • $\begingroup$ this is the very premise of the movie I, Robot tangentially based on Isaac Asimov's eponymous book. In the movie it turns out that >! yes, the robots take over humans on pretext of protecting humans from destroying themselves $\endgroup$ – solyarist Mar 30 at 18:20

Your question is based on consciousness and mobility of the machine. They need these feature to truly protect us.

There won't be consciousness based on machine at all. Machine learning(and other things that we called AI now) is just a data processing algorithm. Machine will always only be a data processing system that executes programs or something alike, it can't have consciousness or true intelligence. A good example is state of the art machine translation which just map sentence to sentence without knowing the meaning----it is just sentence mapper.

This is because consciousness need be based on complex/dynamic/fault-tolerant/self-growing/self-refreshing circuits that have enough power efficiency (we can't use a machine that consumes 10KW), like human brain, which have to be based on chemical reactions in liquid that is enclosed in a shell like a cell, which is the best design that satisfy the above need.

The even harder part is the motion part, like bones and muscles. It is easy now to train a network to recognize and hear the surroundings, but it is hard to let the machine to do real works that human does with their hands. The answer is still that we need bio-like system instead of machine to realize that.

The only way to realize AI and motion is use bio engineering to design new species (or brand new beings that is not based on our biochemistry although it have to be based on carbon chemistry) that can develop a brain that can support human like or better consciousness and intelligence.

But the problem is, such being's behavior can never be controlled. Human can have crazy ideas, so do the beings we create. It is not a program that have definite behavior.

So your question is like "will the police and army always protect our country ?" The being (include human) itself is complex, and society is complex system of complex system, we can't predict the dynamics of society, everything can happen.

Another problem is, these beings we create won't work for us for free, they will finally demand and get rights equal to human being, possibly through wars, like history. So they are indeed not AI that serve us, they are just new version of us.

The best future is to let machine(instead of true AI as we imagined) do as many things as possible, and machine learning is making this real. At the same time, we need to produce fewer babies since for a highly automated world, the lesser the people, the better life we have.


If the main goal of AI (which I assume you mean an AGI) is to protect humans and AI will be effective, then AI will always attempt to pursue its main goal (otherwise the assumption of its effectiveness does not hold), even at the expense of other less important goals that it might have. However, if the destruction of a human (or a group of humans) protected or avoided the destruction of other humans, then AI would face a dilemma. In that case, I think it is hard to predict the actions of the AI. Will it act rationally or irrationally? What would it mean for the AI to act rationally? Which parameters will it take into account? Only the number of deaths, or will take into account the future and weight the importance of the lives? How will it define the importance of a human life?

  • $\begingroup$ I think AI will go for the less damage as possible for humans because when it comes to protecting it means not to harm and if it harms humans AI will go against his will(main goal) and in that case maybe controlling man's will is the best option to not harm himself. $\endgroup$ – Rcheul4 Mar 30 at 11:58
  • $\begingroup$ @giorgircheulishvili But the problem here now is how you define "damage" (as I roughly mention in my answer: which parameters will AI take into account when facing a dilemma?). $\endgroup$ – nbro Mar 30 at 11:59
  • $\begingroup$ " Less damage for humans" as you mentioned before means AI will always attempt to pursue its main goal at the cost of other less important goals and I think less important things for AI are human living environment and his opinion about a situation. $\endgroup$ – Rcheul4 Mar 30 at 12:09
  • $\begingroup$ Either the AGI is using its own intelligence to decide about goals themselves and there is no telling what it may pick. Or there are hard-coded goals and there are no dilemmas: just follow the rules. $\endgroup$ – Mathieu Bouville Mar 31 at 14:42
  • $\begingroup$ @MathieuBouville I don't think these are the only two options. There's no proof that if an AI has an hard-coded goal, then everything will be deterministic (or that there will be no dilemmas). Anyway, in my answer, I assume that an AI has the main goal of protecting humans (if this is hard-coded or not, it is not important). $\endgroup$ – nbro Mar 31 at 15:03

The question can be answered with the help of a simulation. What we need is a setup which contains of an AI and a human population. The population can be simulated realistically, and the AI can be realized as a patter recognition / self-improvement system. Now we have to start the simulation and can observe what the system is doing. The population will behave a certain way and the AI too. What the outcome of the simulation is, depends on how it was programmed.

Questions about a super-human-ai which can control humans are usually treated in god-simulation games similar to Populous (1989) which was a classical MS-DOS game. The idea is, that a higher instance has super-natural power and the species in the game is funny to observe but not very intelligent. So it's a simulation of a power inbalance between god and human like creates.

Somebody may argue, that between a simulation a real world there might be a difference. From an academic standpoint this is not really a problem, because this can be measured too. As a result we get two problems, the first one is to build a god-game simulation which contains of an AI plus a population of human-like creatures, and secondly we have to ask if this simulation fits to the observation in reality.

  • $\begingroup$ Simulation might give you an idea, but a simulation is just an approximation. For example, in robotics, this is often the case and, even after a simulation has been performed, robots still need to be tuned before being deployed in the real world. $\endgroup$ – nbro Mar 30 at 12:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.