Couldn't an AI cheat when trying to follow its goal?

Question

Hello, I was reflecting about what implications might building a strong AI have and I came across some ideas which I find disturbing, I'd love to have some external thought on that :

1) If we ever managed to create an AI say nearly as smart as a human, It would probably have been programmed with some concrete goals, as the AIs we are programming right now : Reinforcement learning allow an agent to try and increase a "reward" variable, regression is all about getting closer to a certain goal function etc..

But then a strong AI, which would undoubtedly be able to understand how it is built, just as we understand (partly at least) how our brains work, because it would be as smart as its creators and we don't tend to build machines that are as hard to understand as brains.

Then couldn't such an agent figure out the best way to achieve its goals would actually not be, say, pleasing and protecting the humans like we would've wanted it to do, but to get control of its own program and maximize whatever reward it was set to pursue ? Just as we could decide to branch electrods to our brain if we were able to find out how exactly our brain was built.

I really don't see how this scenario could ever be avoided if we were to build such an AI, apart from finding a perfect security preventing anyone from accessing the code of the said AI (including itself).

2) On the same note, I also wondered, could it try to not only satisfy its goals by "cheating" (updating its reward variables for example) but also to change itself, or commit suicide ? After all, we humans have never been able to figure out what goals we were meant to pursue (by that I mean what reward variable in our brain we are trying to increment), and many philosophers reflecting upon that matter thought about death as an escape from once goals. So my question is : could it try to change its code or kill itself ?

I have other questions and thoughts I would like to discuss, but I think this a good start, to test whether I'm in the right place for this kind of discussion.

Looking forward to your thoughts.

Answer 1

These are two good questions. To address the first, this is a known problem in AI safety and is called "Wireheading". There is currently no known solution other than to somehow prevent an AI agent from being able to alter its internal state in this fashion. If you are interested in this area in general then I recommend this paper as an entry point into the literature.