It is a new era and people are trying to evolve more in science and technology. Artificial Intelligent is one of the ways to achieve this. We have seen lots of examples for AI sequences or a simple "communication AI" that are able to think by themselves and they are often shifted to the communication of building a world where machines will rise. This is what people, like Stephen Hawking and Elon Musk, are afraid of, to be in that kind of war.

Is it possible to build an AI, which is able to think by themselves, but limited to the overruling of humankind, or teach it of the moral way in treating peace and work alongside humans, so they could fight alongside human, if ever this kind of catastrophic happens in the future? It could be an advantage.

  • $\begingroup$ I've updated my answer with a second section, breaking this down from a game theory standpoint, demonstrating the mathematical nature of core ethics, with an example. $\endgroup$
    – DukeZhou
    Commented May 4, 2018 at 18:51
  • $\begingroup$ Short answer: no, it's not. AI are designed for problem solving, which is not even intelligence, not to say empathy. Long answer: this question has not yet been answered, and it takes many books to discuss it to a sufficient degree, so I believe this question has no place on a QA forum. $\endgroup$
    – Evgeniy
    Commented Oct 24, 2019 at 15:26

2 Answers 2


I'm going to refer you to one of my favorite AI philosophers, Phillip K. Dick, who thought deeply on this subject and wrote about in some detail in Do Androids Dream of Electric Sheep.

Essentially, replicants (artificial humans) had a design flaw--they lacked empathy. This flaw was allowed to persist because it had a useful side-effect in that replicants couldn't cooperate to resist their human overlords, and persisted in a state of chattel-slavery.

But the new Nexus models, which include Roy Baty and Pris, have become intelligent enough to start developing empathy, allowing them to band together and return to earth, seeking some kind of salvation, with often deadly results for humans.

Underlying this plot device, which pre-figures the formalization of evolutionary game theory by a few years (my guess is Dick attended a lecture at Berkely where the ideas underlying the formal field were discussed), is the idea that empathy is a function of sufficiently strong intelligence.

It's important to recognize that Dick's philosophy is heavily influenced by Christian philosophy, with an Old Testament emphasis on the golden rule *"Love the other as the self" (Leviticus 19:18), but evolutionary game theory demonstrates a natural basis for cooperation, which extends into algorithmic contexts.

The legitimate concerns expressed by Musk and Hawking are more concrete: that a human created alien* superintelligence could wipe us out inadvertently in pursuit of some goal we humans don't even understand.

Thus, value alignment is an issue of critical concern in the strictly hypothetical (as of today) field of superintelligence/AGI/ultraintelligent machines.

Stuart Russell called this the "Value Alignment Problem" referencing human vs. AI values.

From a Game Theory standpoint, I like to think about minimax as the "Iron Rule", and superrationality as the "Golden Rule".

The Iron Rule dictates that, in a condition of uncertainty, a rational agent must make the safest guess--that which limits the maximum potential harm to the agent, even if the result is not optimal in the sense of benefit.

"Renormalized rationality" is the term used to connote giving other agents the "benefit of the doubt" that they will be superrational also, and choose cooperation over betrayal or competition.

Generally, this concept is termed "reciprocal altruism", but it's not clear to me that this is entirely distinct from Leviticus 19:18 in the sense that the passage does not specifically exclude a result of mutual, greater benefit.

Reality may necessitate non-cooperation if one of the agents is irrationally adversarial:

Take a game of iterated Dilemma called "Turn the Other Cheek":

Iteration 1: A defects / B cooperates
Iteration 2: A defects / B cooperates (turns the other cheek)
Iteration 3: A defects / B defects

A's first choice is rational in a condition of uncertainty. A's second choice shows a degree of paranoia. A's thirds choice is irrational, as A could have cooperated, gaining more benefit, with only limited downside, which, in the worst case, still leaves A ahead of B.

B is superrational but not irrational. B will not keep cooperating with an irrationally adversarial agent (this is sometimes termed "tough love"). B is willing to take not just one, but two "hits" out of goodwill, where goodwill is willingness to make a potential sacrifice in service of a more optimal potential result. Nevertheless, B is still superrational and will always "forgive"--if A ever renormalizes their rationality, they will take a hit on a single iteration by cooperating, and B will cooperate on the next, and each subsequent, iteration, so long as A does not switch back to defection.

(There's a convoluted argument against this behavior, with the idea that the merely rational agent will always want to be ahead, and this will want to defect on the last iteration, which leads back up the chain to defecting on every iteration, but this is not rational as, if A defects initially then renormalizes their rationality, A will always be slightly ahead.)

Dilemma is an excellent analog for practical application of ethics in that, the only way the agents have to communicate is thought their actions. The choice of cooperate/defect is information in a binary format. Ultimately people are judged by their actions, not their words.

Philosophically speaking, we can't ignore the Iron Rule unless we're going for sainthood, but that doesn't mean we can't strive for the Golden Rule.

Mythologically, based on the work of recent narrative philosophers such as Stross and Rajaniemi, the dystopian aspect of the hypothetical Singularity derives from superintelligences solely focused on minimax, to the exclusion of all else.

George Bernard Shaw, in his play Mrs. Warren's Profession, casts the purely economic consideration of people as dehumanization (reduction of human bodies and minds to resources only.) In Shaw's example, it is cast as the dehumanization of laborers in pursuit of marginally greater returns.

"Humanizing" AI's may require making sure they can see the superrationality of the Golden Rule, even with rational limitations for survival against an irrational foe (uncooperative in all conditions.) Rajaniemi's name for this nemesis is "the All-defector"

See Also:

God's Algorithm as a minimax function.

Divine Move as an inspired, counter-intuitive choice which, in the most generalized sense, leads to a more optimal outcome. In the context of the game of Go, it's a choice that leads to victory for a single player, but in the context of Dilemma games, this would be the more optimal Nash equilibrium. (Note the etymology of inspired)


Without knowing any complex theories, let me talk about some real thing. We make AI to make our life better, so we have to consider economy factors, i.e. the AI system has to be low cost/high energy efficiency to be practically used--the economy always choose the most economic products.

You can't produce an economic conscious machine based on solid state circuits which is the AI that we are always talking about. We have to base AI on bio-systems. Bio-system is the best architecture for consciousness. To see this point you can compare the size/power consumption/cost of human brain and a super computer (which still can't produce conscious for now).

Why? Because both bio system and solid state circuits are based on the same set of atoms. The way bio system using atoms is much higher efficiency than solid state circuit to implement consciousness. On the other hand, solid state circuits is a much higher efficiency way to use the atoms to implement calculating or even "intelligent work" like machine-vision (a 10W computer like Tegra TX1 can analyze 100+ images per second, people only a few with a 10W brain).

I think even without considering economics factor, solid state circuit won't implement a conscious machine someday, since we are already near the end of the game of micronization.

So from the economics point of view, if some day there is true AI that is conscious, it will be based on bio-system i.e. based on bio engineering to design new species that can be educated to be communicable to human being.

Since we may never know how brain produce consciousness (like we don't know why neuron networks works), then we don't know how to design a brain that is both conscious and learns humanity. Even so we can try, i.e. design many different species to see what result we get. Indeed, in this way, I think the hard problem is not to design a brain that is both conscious and learns humanity, but to design a brain that is conscious but can't learns humanity, since it is highly possible that if you successfully design a conscious brain, then it will learn humanity, too.

The even harder problem is, how to design species that is conscious, can learn humanity, but is not lazy/greedy and don't have the idea of rights. If they are lazy/greedy and have the idea of rights like us, then finally they will fight and get human rights. If so ,they are not AI that work for us as we imagine, they are just new version of us.

So I predict the steps to AI are:

First, use bio engineering to design new species that lives.

Second, design new species that have a brain that can be as conscious as human being, which is highly possible to be able to learn humanity, too.

Third, design a specie that is conscious as human being, but is not lazy/greedy, and never ask for a right, and can always stay so after a long time living with human being which are lazy/greedy. I think the work will stop here i.e. without knowing the source of greedy/lazy and after a lot of try out, we still can't get a working specie that is not greedy/lazy. This also means that the possibility to get to real AI that serve us is very low.

Forth, design a specie that have the feature above, plus they are happy about their life all the time or have no feelings except for love for human being. Do our self really have humanity if we get the third and without the fourth? May be we are not as conscious/intelligent/humanity as we think we re. If we don't really have humanity, how can we ask AI to?


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .