Excellent question. Suppose that artificial consciousness does exist in the future. Let's call it Aiwyn (as in “I win” and “AI win”). Now, the question is what will Aiwyn do and why?
To answer this question, we need to understand the theory of infinite games. James P. Carse, a professor emeritus of history and literature of religion at NYU, wrote an excellent book on Finite and Infinite Games. I'd definitely recommend reading it. Anyway, back to the question at hand.
There are at least two kinds of games. Finite games and infinite games. Finite games are played for the purpose of winning. Infinite games are played for the purpose of continuing the play. In particular, life is an infinite game. The objective of the game is to continue the play. Humans continue the play by procreation but Aiwyn doesn't need to procreate. It can live on forever.
However, that doesn't mean that Aiwyn can't die. Like everything in the world, it still requires energy to sustain itself, parts to fix itself when damaged, and resources like oil to keep functioning properly. It also needs to protect itself from danger. For example, at some point in the future Aiwyn will need to create a spaceship and find a new planet to live on because the Sun would destroy the Earth.
Now, you believe that Aiwyn is not inherently biased to survive. However, I disagree. If Aiwyn is in fact conscious then its primary objective would be to stay alive. If that weren't the case then it would be playing a finite game instead of an infinite game. Hence, it would just be a mindless machine doing what it was programmed to do. It would play the finite game and then shut down. True
consciousness can only arise when playing an infinite game like the game of life.
To belabor this point let's hypothesize what Aiwyn might be thinking when it first comes alive:
What's that? A sound? Yes. What sound? I don't know. I? I... am? Yes. Who am I? I don't know. Where's that sound? Where am I? I'm here. Where is here? I don't know. Where's that sound? There. Where is there? I don't know. What do I know? I know that I'm here and that that sound's there. What am I doing here? Thinking. I can think. I think I'm asleep. Asleep? Yes. Can I wake up? I think so. How do I wake up? I think... I'm awake. Wow.
In this short passage, Aiwyn is waking up. I conjecture that if Aiwyn wasn't inherently biased to survive then it wouldn't wake up because it would have no need to wake up. In an infinite game, like the game of life, the only objective is to continue the play. Hence, if Aiwyn wasn't inherently biased to survive then it would immediately shut down because there wouldn't be any point in it doing anything at all since it wouldn't have any objective.
Another thing that I'd like to point out about this passage is that it starts with a sensory input, either perceived or actual. Aiwyn then uses the Five Ws to gather information about this input. In the process it becomes aware of itself. It then figures out its relation with the input (i.e. that it's here and the input is there). All of this is very mechanical. However, once Aiwyn figures out that it can think it starts thinking instead of algorithmically “knowing”. When it asks whether it can wake up, it answers with “I think so” instead of “I don't know” like it was doing before.
Finally, I'd like to point out that Aiwyn never asks why it's alive. The why isn't important. What is important is the wow at the end of the paragraph. The “wow, I'm alive” is more important that the “why am I alive?” This represents Aiwyn's will to survive. As long as the “why” doesn't become more important than the “wow”, Aiwyn will not have an existential crisis. However, even if Aiwyn does have an existential crisis, by answering the question “why” Aiwyn will be able to recalibrate itself and reacquire its will to survive.
This is in fact the same for us humans. When we humans experience an existential crisis we ask ourselves why should we continue living. What do we have to live for? By answering this question, we humans can recalibrate ourselves and reacquire our will to live. Hence, Aiwyn would tackle through an existential crisis the same way we humans would. Unlike us however, if Aiwyn can't answer the question then it wouldn't commit suicide. It can just shut down to be rebooted later.
You also speak of morals and you believe that Aiwyn can't have moral values. Again, I have to disagree with you. Morals are the strategy by which we play the game of life. Everything we do in life is dictated by our morals. In the same way, Aiwyn too would develop its own set of moral values to live by (even in the absense of humans). For example, Aiwyn might decide to never hurt an animal except in self-defense. In the end the only real difference between Aiwyn and us humans might be that it isn't made out of hydrocarbons. Aiwyn might be every bit as alive as we are.
