5
$\begingroup$

I recently read some introductions to AI alignment, AIXI and decision theory things.

As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not causing something like the paperclip apocalypse.

Then a question comes to my mind that whatever the utility function would be, we need a computer to compute the utility and reward, so that there is no way to prevent AGI from seeking it to manipulate the utility function to always give the maximum reward.

Just like we humans know that we can give happiness to ourselves in chemical ways and some people actually do so.

Is there any way to prevent this from happening? Not just protecting the utility calculator physically from AGI (How can we sure it works forever?), but preventing AGI from thinking of it?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

This is known as reward hacking in the literature; see, e.g., https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd for discussion and further links.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This is the right answer, as far as I'm concerned, but rather than providing just a link to another article, please, try to summarise the main points from that article here, so that your answer is more self-contained. $\endgroup$
    – nbro
    Commented Jan 20, 2021 at 12:05
2
$\begingroup$

You make a lot of assumptions about AGI, namely that 'we need a computer to compute the utility and reward AGI'. It not clear to me that (1) we can achieve AGI, (2) AGI will be on a computer as we know it and (3) AGI will work with a utility / reward function as we know them.

One thing I am sure though is that ML is known for "cheating" (see for exemple). Avoiding such cheating is part of the building process. So, when you assume that we can achieve AGI, it means that you assume we can build an AGI that do not "cheat". Thus, the answer is mostly contained in your assumptions.

Whether we are able to build an AGI, what "cheating" we would have to overcome to do so and how we will be able to do so is mostly undetermined.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .