Questions tagged [reward-hacking]

For questions related to reward hacking, i.e. when an agent manipulates the reward system to get a lot of reward in unintended ways. The paper "Concrete Problems in AI Safety" by Dario Amodei et al. provides more details about this topic.

Filter by
Sorted by
Tagged with
1 vote
0 answers
89 views

Has there been an instance of an AI agent breaking out of its sandbox?

There have been instances of agents using edge cases like bugs in physics engines, repetitive behavior in games or word repetition in text prediction to cheat their reward function. However, these ...
2080's user avatar
  • 121
5 votes
2 answers
191 views

How can we prevent AGI from doing drugs?

I recently read some introductions to AI alignment, AIXI and decision theory things. As far as I understood, one of the main problems in AI alignment is how to define a utility function well, not ...
user3584499's user avatar