What are proxy reward functions?

Question

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.

... the observed proxy reward function is the approximate solution to a reward design problem

(source: Inverse Reward Design)

But I have trouble getting how they fit the overall reward objective and got confused by some examples of them. I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from this post, made me question that.

Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...

1. What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?

2. In the examples above, how would high discount rates form a proxy reward?

I'm also curious about how they are used as a source of multiple rewards

Answer 1

In the paper that you cite, Inverse Reward Design (2017), the authors actually define what they mean by "proxy reward function".

We formalize this in a probabilistic model that relates the proxy (designed) reward to the true reward

So, the proxy reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word "proxy" is thus consistent with the general usage of the word in computer science, i.e. a "proxy reward function" is a reward function that is used instead of the intended (optimal) reward function.