Conjecture: regardless of the initial reward function, one of the winning strategies would be to change the reward function to a simpler one (e.g. "do nothing"), thus getting a full reward for each passing unit of time. For such an agent, the only priority would be to prolong its existence (to maximize the overall reward collected).
- The notion of externally defined reward function is incompatible with the concept of self-adjusting AGI.
- Any AGI will always settle on self-preservation as its only goal
- It is therefore impossible to create an AGI with benevolence towards humans "build-in". Instead, the problem of AI alignment should be reformulated in terms of "what changes to the environment (the physical reality that AGI shares with humans) would irreversibly tie wellbeing of humanity to AGIs existence".
- Since any "kill switch" or similar artificial measure is not irreversible and can be overcome by a super-intelligent agent, the only way to tie AGI existence and human wellbeing is the modification of laws of physics, logic, and reasoning. Which is impossible.
- AI alignment is impossible
What flaws do you see in this line of reasoning?