0
$\begingroup$

Hello, I was reflecting about what implications might building a strong AI have and I came across some ideas which I find disturbing, I'd love to have some external thought on that :

1) If we ever managed to create an AI say nearly as smart as a human, It would probably have been programmed with some concrete goals, as the AIs we are programming right now : Reinforcement learning allow an agent to try and increase a "reward" variable, regression is all about getting closer to a certain goal function etc..

But then a strong AI, which would undoubtedly be able to understand how it is built, just as we understand (partly at least) how our brains work, because it would be as smart as its creators and we don't tend to build machines that are as hard to understand as brains.

Then couldn't such an agent figure out the best way to achieve its goals would actually not be, say, pleasing and protecting the humans like we would've wanted it to do, but to get control of its own program and maximize whatever reward it was set to pursue ? Just as we could decide to branch electrods to our brain if we were able to find out how exactly our brain was built.

I really don't see how this scenario could ever be avoided if we were to build such an AI, apart from finding a perfect security preventing anyone from accessing the code of the said AI (including itself).

2) On the same note, I also wondered, could it try to not only satisfy its goals by "cheating" (updating its reward variables for example) but also to change itself, or commit suicide ? After all, we humans have never been able to figure out what goals we were meant to pursue (by that I mean what reward variable in our brain we are trying to increment), and many philosophers reflecting upon that matter thought about death as an escape from once goals. So my question is : could it try to change its code or kill itself ?

I have other questions and thoughts I would like to discuss, but I think this a good start, to test whether I'm in the right place for this kind of discussion.

Looking forward to your thoughts.

$\endgroup$
0
$\begingroup$

These are two good questions. To address the first, this is a known problem in AI safety and is called "Wireheading". There is currently no known solution other than to somehow prevent an AI agent from being able to alter its internal state in this fashion. If you are interested in this area in general then I recommend this paper as an entry point into the literature.

$\endgroup$
0
$\begingroup$

According to the hypothesis, a human level AI is technical possible and the question is how to protect mankind against the threat. To answer the question we have to take a sidestep into underactuated systems which can be controlled only in parts. For example, the heat control of a house can't influence the outside temperature but can only react to a given situation. If Artificial Intelligence is identified as similar to the outside temperature the question is how to react in such an environment.

There are two possible options. First, it's possible to learn from machine intelligence and from robots or secondly it's possible to ignore it. In the first case the chance is better that the control strategy has a positive outcome. That means, if Artificial Intelligence was understood well, it's possible to control the house internal temperature instead of regulating AI itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.