New answers tagged agi
3
A relatively recent but interesting paper that discusses this topic in more detail is Reward is enough (Artificial Intelligence, 2021) by David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton (so by some of the godfathers of RL, who are all at DeepMind).
Their reward-is-enough hypothesis (RIEH) (page 4) is
Hypothesis (Reward-is-Enough). ...
1
Any possible action, including changing the reward function, would be evaluated through the initial reward function. In order to avoid the scenario you described, a reward function needs to disincentivize changes to itself by giving those the lowest possible reward.
-2
AIXI is important, the reinforcement learning we already see is a smaller classical version of the full intractable model, that one would require quantum computation to see it come to fruition.
Top 50 recent answers are included
