Conjecture: regardless of the initial reward function, one of the winning strategies would be to self-modify to change the reward function to a simpler one (e.g. "do nothing") thus getting full reward for each passing unit of time. for such an agent, the only priority would be to prolong its existence (to maximize the overall reward collected).


  • The notion of externally defined reward function is incompatible with the concept of self-adjusting AGI.
  • Any AGI will always settle on self-preservation as its only goal
  • It is therefore impossible to create AGI with benevolence towards humans "build-in". Instead, the problem of AI alignment should be reformulated in terms of "what changes to the environment (physical reality that AGI shares with humans) would irreversibly tie wellbeing of humanity to AGIs existence".
  • Since any "kill switch" or similar artificial measure is not irreversible and can be overcome by a super intelligent agent, the only way to tie AGI existence and human wellbeing is modification of laws of physics, logic and reasoning. Which is impossible.
  • AI alignment is impossible

What flaws do you see in this line of reasoning?

  • 1
    $\begingroup$ I guess there's a typo here "...would be to self-modify to change...". $\endgroup$
    – nbro
    Jul 1 at 20:53

Any possible action, including changing the reward function, would be evaluated through the initial reward function. In order to avoid the scenario you described, a reward function needs to disincentivize changes to itself by giving those the lowest possible reward.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.