11
$\begingroup$

This question stems from quite a few "informal" sources. Movies like 2001, A Space Odyssey and Ex Machina; books like Destination Void (Frank Herbert), and others suggest that general intelligence wants to survive, and even learn the importance for it.

There may be several arguments for survival. What would be the most prominent?

Improve this question
$\endgroup$
4
  • $\begingroup$ @DukeZhou Would you have public documents on the topic? $\endgroup$
    – Eric Platon
    Oct 15, 2018 at 4:47
  • 1
    $\begingroup$ Not as yet. (It's going to be a long process as our development cycle is very slow. I also probably shouldn't have said mentioned AGI in this context because what I'm really looking at is more of a generalized utility function that mitigates risk, not a super-intelligence!) Wanted to update you, but probably best if I delete my original comment for being overly-optimistic/misleading in this regard. $\endgroup$
    – DukeZhou
    Oct 15, 2018 at 16:48
  • 1
    $\begingroup$ Essentially, my thought was that a general economic utility function would form a good basis for an AGI by providing a prime motive. (The idea is that general intelligence is just a degree of capability re: an infinitely expanding set of problems.) It would relate to survival instinct in the sense that the automata is driven toward optimal decisionmaking in the context of maintaining and bettering it's condition, (though not to the point of mindless expansion which could be harmful to its "ecosystem), where every decision is ultimately regarded as economic. $\endgroup$
    – DukeZhou
    Oct 18, 2018 at 3:08
  • 1
    $\begingroup$ (if you have an "online venue" where those topics can be discussed, that'd be great---I think we are getting kicked off this site) $\endgroup$
    – Eric Platon
    Oct 18, 2018 at 8:52

3 Answers 3

Reset to default
6
$\begingroup$

Steve Omohudro wrote a paper called Basic AI Drives that steps through why we would expect an AI with narrow goals to find some basic, general concepts as instrumentally useful for their narrow goals.

For example, an AI designed to maximize stock market returns but whose design is silent on the importance of continuing to survive would realize that its continued survival is a key component of maximizing stock market returns, and thus take actions to keep itself operational.

In general, we should be skeptical of 'anthropomorphizing' AI and other code, but it seems like there are reasons to expect this beyond "well, humans behave this way, so it must be how all intelligence behaves."

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I assume Omohudru uses "maximize returns" in the context of minimax, where mitigation of downside is part of the equation. From my perspective, "win else survive" is a basic extension of minimax in cyclic games. $\endgroup$
    – DukeZhou
    Apr 22, 2017 at 20:46
5
$\begingroup$

The concept of 'survival instinct' probably falls in the category of what Marvin Minsky would call a 'suitcase word', i.e. it packages together a number of related phenomena into what at first appears to be a singular notion.

So it's quite possible that we can construct mechanisms that have the appearance of some kind of 'hard-coded' survival instinct, without that ever featuring as an explicit rule(s) in the design.

See the beautiful little book 'Vehicles' by the neuroanatomist Valentino Braitenberg for a compelling narrative of how such 'top down' concepts as 'survival instinct' might evolve 'from the bottom up'.

Also, trying to ensure that intelligent artefacts place too high a priority on their survival might easily lead to a Killbot Hellscape.

Improve this answer
$\endgroup$
2
$\begingroup$

I will answer by quoting the book The Myth of Sisyphus:

There is only one really serious philosophical question, and that is suicide.

So, probably, we need some degree of a survival instinct if we don't want our AGIs to "terminate themselves" (whatever that means), whenever they get existential.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .