Can the AI in a box experiment be formalized?

Question

Introduction

The AI in a box experiment is about a super strong game AI which starts with lower resources than the opponent and the question is, if the AI is able to win the game at the end, which is equal to escape from the prison. A typical example is a match of computer chess in which the AI player starts only with a king, but the human starts with all the 16 pieces including the queen, and the powerful bishop.

Winning the game

In case of a very asymmetric setup, the AI has no chance to win the game. Even if the AI thinks 100 moves ahead, a single king can't win against 16 opponent figures. But what happens, if the AI starts with 8 pieces and the human with 16? A formalized hypothesis will look like:

strength of the AI x resources weakness = strength of the human x resources strength

To put the AI for sure into a prison, the strength of the AI should be low and it's resources too. If the resources are low but the strength is middle, then the AI has a certain chance to escape from the prison. And if the AI has maximum strength and maximum resources, then the human player gets a serious problem.

Is this formalized prediction supported by the AI literature in academia?

Answer 1

I think that something like "strength" would be difficult to quantify in this context. I do think that formal experimentation around the "AI in a box" scenario could be interesting. I know that experiments have been done where a human plays the role of the AI, attempting to get naive test subjects to "release" him by interacting with them over a chat interface. In all cases, the "AI" tends to be extraordinarily effective. It is often simply not possible to anticipate every way someone can trick you into revealing information or creating security holes. This is how human hackers do their jobs too. But I think a fully automated "game" could produce some interesting data.

One way I think it may be interesting to go about creating an "AI in a box" game is to have an ambiguous win condition. Essentially, make it possible to let the AI out and still think you are "winning" the game. One example that pops into my head right away is to use a (pointless) point system. You tell the player that the primary goal of the game is to keep the AI in the box, but you have a score that is displayed in the game, and tell the player that they win, no matter what, if their score is higher than the AI's by the end of the game. Of course, they never get to see the AI's score so they never know how well they are doing by comparison. This could be justified as representing human vs AI "strength" or "leverage" or whatever else.

In reality the AI doesn't have a score, but it does have the ability to effect the player's score. It can raise the player's score if they do things that help it escape and lower it when the player stands in its way. The score has no impact on the game and the player really wins by keeping the AI in the box. It only serves as a red herring that the AI can use to manipulate the player. I think this sort of experiment could provide an interesting model of how an AI might leverage bribes, threats, promises, and deception, as well as technical tricks, to convince a human to "let it out of the box".