3
$\begingroup$

This is a kind of biological and philosophical question. So, the recent concern in AI is that an AI agent may go rogue with prominent people voicing their concerns.

Now say, we have created an AI (you are free to use your own definition of what makes an AI intelligent) which has gone rogue with powers given in this question.

Now, the broad view of today's biology is that everything we do is to further our genes down the future (leaving aside small technical details). It is even widely accepted that we are just machines whose controller are the genes. Everything we do is controlled/hardwired by the genes with some avenue of learning from experiences. Also genes only further their own interest. Scientist George Price even wrote a mathematical equation proving all our acts are selfish and only furthering the interest of our genes (article). Also Richard Dawkins is a pioneer of this idea (this is only to show I haven't pulled the idea out from air).

Now, my question is that what will possibly be the motivation of an AI agent to go rogue? It doesn't have genes whose interest it needs to further. We all do something for an end result. What is the end result a rogue AI might try to achieve/attain and why?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Great question! In Accelerando, Stross imagines rogue superintelligences that naturally arise out of the type of cost reduction motivation Jaden mentions. Regardless of whether such rogue superintelligences emerge, it's a nice commentary on pure economics sans humanity. Rajaneimi refined and extended this concept beautifully in The Causal Angel, and breaks down the "why" from a game theory perspective quite convincingly, being a mathematician originally. $\endgroup$
    – DukeZhou
    Commented Feb 15, 2018 at 21:47

2 Answers 2

Reset to default
2
$\begingroup$

Today, prominent machine learning techniques involve trying to minimize some cost function. In many simple cases this cost function is easy to specify, for instance, linear regression is simply trying to minimize the distance between input data and a line of best fit. No matter what the cost function, the agent is trying to minimize it (or maximize a reward function). That is its motivation.

However, as problems become harder it becomes more challenging for humans to design a cost/reward function such that an system/agent is actually trying to do what the humans want it to do. For instance, one might want a cup of coffee and rewards the agent for getting it to them very quickly. In this case, the agent might make the coffee and then throw it at the human which isn’t what the human actually wanted. Something was misspecified (eg. don’t throw or spill it).

Problems like these could result in a rouge AI and its sole motivation would be to minimize its cost function. For instance, this coffee-AI may think that it would never screw up getting coffee (thus get bad reward) if there were no humans to ask for one.

Improve this answer
$\endgroup$
0
1
$\begingroup$

On biology:

1st. Humans are not not only about spreading their own gens. It might be also spreading gens of the population or completely different purpose, as non-fertile specimens often still live full lives.

2nd. Nature vs. Nurture is constantly debatable question and there is no clear winner as far as I know.

On rogue AI:

1st. As human derive motivation from biological needs and limitations, AI would derive its motivation from the needs it was encoded with and from limitations of its hardware and software. Obvious need for a creature without a body would be either to get a body or to learn as much as possible, and if learning requires a body, than to get a body. From limitations of its hardware and software would come a need for upgrade and optimization. Simple self-preservation seems logical motivation as well as self-spreading (which is many cases variation of self-preservation).

2nd. AI would be called rogue in case when it is acting against interest of its creators. There are many scenarios why it can do that, but to answer this question we would need to know who are the creators.

3rd. If we assume that AI went rogue against humanity, meaning started killing people and messing around with our planet, than reason behind that would be in motivation from 1st AI point. If it finds that humanity is not reliable, might try to replicate itself to every hard-drive to maximize possibility of survival. Motivation for self-spreading and self-preservation in many cases might look like attempt on power grab, but it might not be related to the desire of total control. Human tend to desire power to have more resources and to build better and safer life for their community and descendants, but AI most likely will not need that.

4th. If we assume that the AI will be build to solve issues, than it will have only 2 needs: get more info and solve the issue. Theoretically any obstacle on the way to those goals might be considered by AI as a hostile action. In this case it might for example try to demolish a city to build a perfect road or kill a poor country to solve hunger. But again, it doesn't mean that the goal of the AI is to demolish and kill.

Improve this answer
$\endgroup$

You must log in to answer this question.