I'm going to refer you to one of my favorite AI philosophers, Phillip K. Dick, who thought deeply on this subject and wrote about in some detail in Do Androids Dream of Electric Sheep.
Essentially, replicants (artificial humans) had a design flaw--they lacked empathy. This flaw was allowed to persist because it had a useful side-effect in that replicants couldn't cooperate to resist their human overlords, and persisted in a state of chattel-slavery.
But the new Nexus models, which include Roy Baty and Pris, have become intelligent enough to start developing empathy, allowing them to band together and return to earth, seeking some kind of salvation, with often deadly results for humans.
Underlying this plot device, which pre-figures the formalization of evolutionary game theory by a few years (my guess is Dick attended a lecture at Berkely where the ideas underlying the formal field were discussed), is the idea that empathy is a function of sufficiently strong intelligence.
It's important to recognize that Dick's philosophy is heavily influenced by Christian philosophy, with an Old Testament emphasis on the golden rule *"Love the other as the self" (Leviticus 19:18), but evolutionary game theory demonstrates a natural basis for cooperation, which extends into algorithmic contexts.
The legitimate concerns expressed by Musk and Hawking are more concrete: that a human created alien* superintelligence could wipe us out inadvertently in pursuit of some goal we humans don't even understand.
Thus, value alignment is an issue of critical concern in the strictly hypothetical (as of today) field of superintelligence/AGI/ultraintelligent machines.
Stuart Russell called this the "Value Alignment Problem" referencing human vs. AI values.
From a Game Theory standpoint, I like to think about minimax as the "Iron Rule", and superrationality as the "Golden Rule".
The Iron Rule dictates that, in a condition of uncertainty, a rational agent must make the safest guess--that which limits the maximum potential harm to the agent, even if the result is not optimal in the sense of benefit.
"Renormalized rationality" is the term used to connote giving other agents the "benefit of the doubt" that they will be superrational also, and choose cooperation over betrayal or competition.
Generally, this concept is termed "reciprocal altruism", but it's not clear to me that this is entirely distinct from Leviticus 19:18 in the sense that the passage does not specifically exclude a result of mutual, greater benefit.
Reality may necessitate non-cooperation if one of the agents is irrationally adversarial:
Take a game of iterated Dilemma called "Turn the Other Cheek":
Iteration 1: A defects / B cooperates
Iteration 2: A defects / B cooperates (turns the other cheek)
Iteration 3: A defects / B defects
A's first choice is rational in a condition of uncertainty. A's second choice shows a degree of paranoia. A's thirds choice is irrational, as A could have cooperated, gaining more benefit, with only limited downside, which, in the worst case, still leaves A ahead of B.
B is superrational but not irrational. B will not keep cooperating with an irrationally adversarial agent (this is sometimes termed "tough love"). B is willing to take not just one, but two "hits" out of goodwill, where goodwill is willingness to make a potential sacrifice in service of a more optimal potential result. Nevertheless, B is still superrational and will always "forgive"--if A ever renormalizes their rationality, they will take a hit on a single iteration by cooperating, and B will cooperate on the next, and each subsequent, iteration, so long as A does not switch back to defection.
(There's a convoluted argument against this behavior, with the idea that the merely rational agent will always want to be ahead, and this will want to defect on the last iteration, which leads back up the chain to defecting on every iteration, but this is not rational as, if A defects initially then renormalizes their rationality, A will always be slightly ahead.)
Dilemma is an excellent analog for practical application of ethics in that, the only way the agents have to communicate is thought their actions. The choice of cooperate/defect is information in a binary format. Ultimately people are judged by their actions, not their words.
Philosophically speaking, we can't ignore the Iron Rule unless we're going for sainthood, but that doesn't mean we can't strive for the Golden Rule.
Mythologically, based on the work of recent narrative philosophers such as Stross and Rajaniemi, the dystopian aspect of the hypothetical Singularity derives from superintelligences solely focused on minimax, to the exclusion of all else.
George Bernard Shaw, in his play Mrs. Warren's Profession, casts the purely economic consideration of people as dehumanization (reduction of human bodies and minds to resources only.) In Shaw's example, it is cast as the dehumanization of laborers in pursuit of marginally greater returns.
"Humanizing" AI's may require making sure they can see the superrationality of the Golden Rule, even with rational limitations for survival against an irrational foe (uncooperative in all conditions.) Rajaniemi's name for this nemesis is "the All-defector"
See Also:
God's Algorithm as a minimax function.
Divine Move as an inspired, counter-intuitive choice which, in the most generalized sense, leads to a more optimal outcome. In the context of the game of Go, it's a choice that leads to victory for a single player, but in the context of Dilemma games, this would be the more optimal Nash equilibrium. (Note the etymology of inspired)
