Why cannot an AI agent adjust the reward function directly?

Question

In standard Reinforcement Learning the reward function is specified by an AI designer and is external to the AI agent. The agent attempts to find a behaviour that collects higher cumulative discounted reward. In Evolutionary Reinforcement Learning the reward function is specified by the agent’s genetic code and evolved in simulated Darwinian evolution over multiple generations. Here too the AI agent cannot directly adjust the reward function and instead adjusts its behaviour towards collecting higher rewards. Why do both approaches prevent the AI agent from changing its reward function at will? What happens if we do allow the AI agent to do so?

Answer 1

Why do both approaches prevent the AI agent from changing its reward function at will?

In RL for optimal control, the reward function is part of the problem formulation. That is, it describes the goals of the agent. Sometimes this is obviously something that should not be under the agent's control, if the reward is a real-world quantity that it should maximise - e.g. the amount of money it makes in profit - then it makes no sense that the agent could arbitrarily declare that the quantity is different to the thing that it observed.

Other times, there is some flexibility, an agent that needs to escape a maze in a short time could have -1 reward per time step inside the maze or -0.1 reward per time step, or +1 reward for escaping with a discount factor applied. However, the flexibility can only go so far before it describes a different problem. Changing the -1 per time step to +1 per time step means that the agent's goal switches from escaping to staying in the maze.

In general, multiplying all the rewards in a MDP by some positive constant does not change a reinforcement learning problem. Sometimes it may be worth doing this scaling to make it easier for a specific approach, such as neural networks, to work efficiently. However, this is not something to put directly under the agent's control, but a hyperparameter like the number of hidden layers in the neural network. As a hyperparameter, usually the reward scaling is very flexible and not something worth spending much effort tuning - unlike the architecture of a neural network.

What happens if we do allow the AI agent to do so?

Unless significant constraints are placed on what is allowed to change, then the agent will get any amount of reward it "wants" by doing anything it "wants", within whatever constraints are placed on changes allowed to the reward function. Typically in RL this would result in an agent that acts more or less randomly whilst getting progressively higher and higher reward on each iteration. Or in other words, an agent that does not attempt to solve any kind of problem.

There are a few special cases where a reward function can be adjusted or learned. One common case is inverse reinforcement learning, where an agent's activities are observed, it is assumed to be solving an MDP-like problem, and you are interested in understanding how it solves it, including what reward function it is using. The reward function must be learned by fitting it to observations of the agent.