11
$\begingroup$

This question stems from quite a few "informal" sources. Movies like 2001, A Space Odyssey and Ex Machina; books like Destination Void (Frank Herbert), and others suggest that general intelligence wants to survive, and even learn the importance for it.

There may be several arguments for survival. What would be the most prominent?

$\endgroup$
  • $\begingroup$ @DukeZhou Would you have public documents on the topic? $\endgroup$ – Eric Platon Oct 15 '18 at 4:47
  • 1
    $\begingroup$ Not as yet. (It's going to be a long process as our development cycle is very slow. I also probably shouldn't have said mentioned AGI in this context because what I'm really looking at is more of a generalized utility function that mitigates risk, not a super-intelligence!) Wanted to update you, but probably best if I delete my original comment for being overly-optimistic/misleading in this regard. $\endgroup$ – DukeZhou Oct 15 '18 at 16:48
  • 1
    $\begingroup$ Essentially, my thought was that a general economic utility function would form a good basis for an AGI by providing a prime motive. (The idea is that general intelligence is just a degree of capability re: an infinitely expanding set of problems.) It would relate to survival instinct in the sense that the automata is driven toward optimal decisionmaking in the context of maintaining and bettering it's condition, (though not to the point of mindless expansion which could be harmful to its "ecosystem), where every decision is ultimately regarded as economic. $\endgroup$ – DukeZhou Oct 18 '18 at 3:08
  • 1
    $\begingroup$ (if you have an "online venue" where those topics can be discussed, that'd be great---I think we are getting kicked off this site) $\endgroup$ – Eric Platon Oct 18 '18 at 8:52
6
$\begingroup$

Steve Omohudro wrote a paper called Basic AI Drives that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals.

For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.

In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there are reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."

$\endgroup$
  • 1
    $\begingroup$ I assume Omohudru uses "maximize returns" in the context of minimax, where mitigation of downside is part of the equation. From my perspective, "win else survive" is a basic extension of minimax in cyclic games. $\endgroup$ – DukeZhou Apr 22 '17 at 20:46
5
$\begingroup$

The concept of 'survival instinct' probably falls in the category of what Marvin Minsky would call a 'suitcase word', i.e. it packages together a number of related phenomena into what at first appears to be a singular notion.

So it's quite possible that we can construct mechanisms that have the appearance of some kind of 'hard-coded' survival instinct, without that ever featuring as an explicit rule(s) in the design.

See the beautiful little book 'Vehicles' by the neuroanatomist Valentino Braitenberg for a compelling narrative of how such 'top down' concepts as 'survival instinct' might evolve 'from the bottom up'.

Also, trying to ensure that intelligent artefacts place too high a priority on their survival might easily lead to a Killbot Hellscape.

$\endgroup$
2
$\begingroup$

I will answer by quoting the book The Myth of Sisyphus:

There is only one really serious philosophical question, and that is suicide.

So, probably, we need some degree of a survival instinct if we don't want our AGIs to "terminate themselves" (whatever that means), whenever they get existential.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.