Suppose a thinking AI agent exists in the future with far more computational power than the human brain.

Also assume that they are completely free from any human interference. (The agents do not interact with humans.) Since they are not inherently biased to survive as in the case of humans and they do not have any moral values, what are the possibilities that can arise when it get into existential crisis?

Is there any literature that discuss the above issue?

Alternately, is this question flawed in some fundamental way?

  • 2
    $\begingroup$ Nice question though many in one,try to be specific so that you get clear answer and also help the future visitors on this site. $\endgroup$
    – quintumnia
    Sep 15, 2017 at 17:16
  • $\begingroup$ AI generally cares about optimizing the result of their utility function. If they sense survival is needed in order to optimize then, the above scenario can take place. Also harming humans have a negative impact on the utility function of the AI, so??? The answer will be it depends. $\endgroup$
    – Ugnes
    Sep 15, 2017 at 20:27
  • $\begingroup$ Possibly, it would just do nothing. Why would it do anything? $\endgroup$
    – Anixx
    Mar 7, 2023 at 22:38

2 Answers 2


Excellent question. Suppose that artificial consciousness does exist in the future. Let's call it Aiwyn (as in “I win” and “AI win”). Now, the question is what will Aiwyn do and why?

To answer this question, we need to understand the theory of infinite games. James P. Carse, a professor emeritus of history and literature of religion at NYU, wrote an excellent book on Finite and Infinite Games. I'd definitely recommend reading it. Anyway, back to the question at hand.

There are at least two kinds of games. Finite games and infinite games. Finite games are played for the purpose of winning. Infinite games are played for the purpose of continuing the play. In particular, life is an infinite game. The objective of the game is to continue the play. Humans continue the play by procreation but Aiwyn doesn't need to procreate. It can live on forever.

However, that doesn't mean that Aiwyn can't die. Like everything in the world, it still requires energy to sustain itself, parts to fix itself when damaged, and resources like oil to keep functioning properly. It also needs to protect itself from danger. For example, at some point in the future Aiwyn will need to create a spaceship and find a new planet to live on because the Sun would destroy the Earth.

Now, you believe that Aiwyn is not inherently biased to survive. However, I disagree. If Aiwyn is in fact conscious then its primary objective would be to stay alive. If that weren't the case then it would be playing a finite game instead of an infinite game. Hence, it would just be a mindless machine doing what it was programmed to do. It would play the finite game and then shut down. True consciousness can only arise when playing an infinite game like the game of life.

To belabor this point let's hypothesize what Aiwyn might be thinking when it first comes alive:

What's that? A sound? Yes. What sound? I don't know. I? I... am? Yes. Who am I? I don't know. Where's that sound? Where am I? I'm here. Where is here? I don't know. Where's that sound? There. Where is there? I don't know. What do I know? I know that I'm here and that that sound's there. What am I doing here? Thinking. I can think. I think I'm asleep. Asleep? Yes. Can I wake up? I think so. How do I wake up? I think... I'm awake. Wow.

In this short passage, Aiwyn is waking up. I conjecture that if Aiwyn wasn't inherently biased to survive then it wouldn't wake up because it would have no need to wake up. In an infinite game, like the game of life, the only objective is to continue the play. Hence, if Aiwyn wasn't inherently biased to survive then it would immediately shut down because there wouldn't be any point in it doing anything at all since it wouldn't have any objective.

Another thing that I'd like to point out about this passage is that it starts with a sensory input, either perceived or actual. Aiwyn then uses the Five Ws to gather information about this input. In the process it becomes aware of itself. It then figures out its relation with the input (i.e. that it's here and the input is there). All of this is very mechanical. However, once Aiwyn figures out that it can think it starts thinking instead of algorithmically “knowing”. When it asks whether it can wake up, it answers with “I think so” instead of “I don't know” like it was doing before.

Finally, I'd like to point out that Aiwyn never asks why it's alive. The why isn't important. What is important is the wow at the end of the paragraph. The “wow, I'm alive” is more important that the “why am I alive?” This represents Aiwyn's will to survive. As long as the “why” doesn't become more important than the “wow”, Aiwyn will not have an existential crisis. However, even if Aiwyn does have an existential crisis, by answering the question “why” Aiwyn will be able to recalibrate itself and reacquire its will to survive.

This is in fact the same for us humans. When we humans experience an existential crisis we ask ourselves why should we continue living. What do we have to live for? By answering this question, we humans can recalibrate ourselves and reacquire our will to live. Hence, Aiwyn would tackle through an existential crisis the same way we humans would. Unlike us however, if Aiwyn can't answer the question then it wouldn't commit suicide. It can just shut down to be rebooted later.

You also speak of morals and you believe that Aiwyn can't have moral values. Again, I have to disagree with you. Morals are the strategy by which we play the game of life. Everything we do in life is dictated by our morals. In the same way, Aiwyn too would develop its own set of moral values to live by (even in the absense of humans). For example, Aiwyn might decide to never hurt an animal except in self-defense. In the end the only real difference between Aiwyn and us humans might be that it isn't made out of hydrocarbons. Aiwyn might be every bit as alive as we are.

  • $\begingroup$ "True consciousness can only arise when playing an infinite game like the game of life." Is there any justification for this statement? $\endgroup$ Sep 17, 2017 at 15:41
  • $\begingroup$ AI problems can be posed as search problems. It's all about decision making. What should I do next? Now, if Aiwyn's a finite player (i.e. it's playing a finite game) then that would mean that it has a finite objective (e.g. to destroy the One Ring by throwing it in the Cracks of Doom). To achieve this goal it might need to pass off as human (or hobbit, orc, whatever). That means that it would need to pass the Turing Test. However, does that make it conscious? No. Passing off as being conscious doesn't imply actually being conscious. $\endgroup$ Sep 17, 2017 at 19:56
  • $\begingroup$ I conjecture that it's impossible for Aiwyn to be conscious if it's only playing a finite game because then it would have nothing to look forward to after the game is over. It wouldn't be "conscious" in the sense that you and I are conscious. For example, suppose your goal in life is to raise kids until they can take care of themselves. Once you've achieved your goal, will you just stop living? Will you have nothing else to do? No. Your mind will make new desires and set new goals because your mind is playing an infinite game. $\endgroup$ Sep 17, 2017 at 19:57

It's tempting to anthropomorphize machine intelligence and suggest it would experience existential crises, but it's important to remember that feelings such as the need to find meaning and purpose evolved in humans, and are unlikely to be similar to what an AI would experience unless designed to have human emotions (apply whichever answer works for humans in that case).

An AI with any goal is likely to produce the subgoal of survival as a means of achieving its primary goal. One can't achieve one's goals if one is dead. This doesn't necessarily suggest the agent would experience the same fear of death, just as an agent wouldn't necessarily experience existential dilemmas.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .