1
$\begingroup$

There have been instances of agents using edge cases like bugs in physics engines, repetitive behavior in games or word repetition in text prediction to cheat their reward function. However, these agents are arguably still contained, as while they explore the extremes of the state space of the simulation they don't expand their action space beyond what is possible in the simulation.

The Pokémon Yellow Total Control Hack shows that in some systems, it is possible to gain full control of a computer by exploiting bugs in the hardware or software (here: memory corruption), enabling the agent to even completely reprogram the system 'from within', just using the normal inputs.

Do you know of similar extreme examples where an AI agent went far beyond what was intended with the simulation environment and expanded its action space?

Improve this question
$\endgroup$
6
  • $\begingroup$ This kind of scenario is still solidly in the realms of science fiction. The current crop of optimising and statistical pattern AIs are not "minds" in any common sense understanding of the word. $\endgroup$
    – Neil Slater
    Jul 2, 2022 at 18:29
  • $\begingroup$ "There have been instances of agents using edge cases like...", can you please provide a reference that supports all your examples? $\endgroup$
    – nbro
    Jul 2, 2022 at 22:20
  • $\begingroup$ @nbro: The first link is to a short video which does demonstrate a game physics exploit being used to gain reward for a task. I think someone has collated a few similar examples on a page somewhere, I'll see if I can find it $\endgroup$
    – Neil Slater
    Jul 3, 2022 at 7:20
  • 3
    $\begingroup$ @nbro: Here's a good list, and it might be the foundation of an answer: docs.google.com/spreadsheets/d/e/… (but as per my comment, the idea of an agent somehow "evolving" or transcending its state/action space to attack the hardware or redefine itself is still far out of reach). $\endgroup$
    – Neil Slater
    Jul 3, 2022 at 7:25
  • $\begingroup$ @NeilSlater I had come across that list in the past and I had completely forgotten about it when I wrote my comment above. Thanks. $\endgroup$
    – nbro
    Jul 4, 2022 at 22:22

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .