5
$\begingroup$

Let's take our standard paperclip maximizer General AI and attempt to obtain precisely one million paper clips, over course of a year, without destroying the universe in the process.

Most maximization directives make the process run-away. As cheaply as possible will crash world economy. As good clips as possible will turn the universe into super-synthesizer that assembles atom-perfect paperclips. Adding a deadline on these maximization processes will probably result in terrorizing the staff into readjusting the deadline, or invention of time travel (after consuming the solar system to invent it.) Minimizing resource usage would likely result in closure of all industry world-wide. You know, the standard horror scenarios.

What about the directive of minimizing AI's influence on the world while completing the task? Would it be safe, or can you spot a runaway scenario where it could result in dire effects?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ it would also be quite diminishing the potential power of AI $\endgroup$
    – k.c. sayz 'k.c sayz'
    Jan 27, 2018 at 20:00
  • 1
    $\begingroup$ @k.c.sayz'k.csayz': The problem is the consequences for it going runaway are way worse than any loss of performance. $\endgroup$
    – SF.
    Jan 27, 2018 at 21:27
  • $\begingroup$ i added an answer that is a response to the above $\endgroup$
    – k.c. sayz 'k.c sayz'
    Jan 28, 2018 at 0:42
  • $\begingroup$ Hannu Rajaniemi wrote an interesting story on this subject, called "Deus Ex Homine" re: benevolent superintelligences, described as recursively self-optimizing algorithms. In this story, the superintelligences continually improve and fix things, but are ungovernable and run wild. It's an interesting take because it looks at the scenario from the opposite angle (i.e. not superintelligence that want to destroy humanity, but improve it. :) $\endgroup$
    – DukeZhou
    Jan 28, 2018 at 22:22

4 Answers 4

Reset to default
3
$\begingroup$

I like mindrime's answer as it identifies fundamental, applicable notions. I'll attempt another short answer from a Game Theory perspective.

Game Theory was founded in the minimax principle. Specifically, maximizing benefit while minimizing potential downside in a condition of uncertainty.

Minimax works very well in contexts with easily definable parameters, such as combinatorial games and procedural optimization, but it gets more tricky in real-world scenarios where too many parameters may be present, resulting in a combinatorial explosion.

Another issue regarding real-world applications arises from the symbol grounding problem. In the paper clip scenario, the benefit sought (the goal) is clearly definable, and can be expressed mathematically. By contrast, the downsides to be avoided are more difficult to define: "Don't destroy the world", "Don't use up resources to a degree that humans suffer", "Don't harm the environment" all rely on language. This is to say they rely on terms that, at present, constitute symbols that cannot be grounded. Thus there is room for misinterpretation that may yield unintended consequences.

"Minimizing impact on the system" (minimizing influence in the world) would be the goal, but how do you guarantee an automata has a clear understanding of what this entails in every possible scenario?

Improve this answer
$\endgroup$
2
$\begingroup$

Telling the system "minimize your interference with the world" while also telling it to "maximize paperclip production" or whatever is interesting on at least one level, and that is this: how exactly does the system quantify "interference in the world"? That seems like an ill-defined notion offhand. But if you could quantify it, then it just becomes one more variable in an optimization problem which is a straightforward notion.

Detecting a runaway process in general is an interesting concept. I am not an expert, but I am betting there is some material in the cybernetics / control theory literature on this topic. It might be as simple as watching some rate-of-change (paper-clips produced per day?) and take the first and second derivatives and look for sharp changes in acceleration or jerk. Other algorithms from the world of anomaly detection might also be applicable.

Improve this answer
$\endgroup$
1
$\begingroup$

From the AI in a box literature, it is argued that even just a text interface with the rest of the world is sufficient for an AI to gain total control.

Or, consider the literature regarding phase state changes / dynamical systems / control theory. I don't know if there is a source that directly argues for this, but its imaginable that since societal systems are so interconnected, a few controllable free parameters of a system might be sufficient to strongly influence the system as a whole.

So no, restricting influence is not a sufficient guarantee of reducing AI risk. A common saying is that if we know the goal of some AI, we can't predict /how/ the AI would achieve some goal since we aren't smart enough, but we can predict the eventual outcome (its success).

Improve this answer
$\endgroup$
0
$\begingroup$

Here you make the wrong assumption that AI will have only one goal at a time. But the same as humans it will have to have in mind many goals all the time it should follow and watch out that newly assigned goal dont conflict with its existing goals.

Your proposition to give ai the goal "minimize impact on world" is simplistic as it would be harmful in some situations.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .