# What are proxy reward functions?

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.

... the observed proxy reward function is the approximate solution to a reward design problem

(source: Inverse Reward Design)

But I have trouble getting how they fit the overall reward objective and got confused by some examples of them. I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from this post, made me question that.

Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...

1. What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?

2. In the examples above, how would high discount rates form a proxy reward?

I'm also curious about how they are used as a source of multiple rewards

• Maybe have a look at ai.stackexchange.com/q/20040/2444, although I think that you know what "proxy" usually means. – nbro Sep 22 '20 at 11:24
• So "proxies" are intermediaries. This helped add clarity into the "what are they" part of the question – mugoh Sep 23 '20 at 5:05
• Please, read my answer below ai.stackexchange.com/a/24074/2444, if you haven't yet. You should accept it, if it answers your question. I read that IRD paper and other related papers, so I'm pretty confident that the answer is correct. If something is still unclear, let me know. Please, take a look at ai.stackexchange.com/help/someone-answers for more details. – nbro Jan 16 at 18:17
• Thanks for pointing this out @nbro. I was also seeking clarity on how high discount rates form proxy rewards and hiw proxies can be used as a source of multiple rewards – mugoh Jan 17 at 4:13
• It's not indeed clear what the article's author meant by "artificially high discount rates". It's not even clear if they are referring to the discount factor of the RL algorithm. Even if that was the case, it's not very clear why that would be a "proxy reward function". The discount factor is the way to weight short-term and long-term rewards, but it's not very clear why one would consider that a "proxy reward function". Maybe this has to do with the "long time horizon" part of that article. Maybe the best thing to do is e-mail the article's author and ask him to answer your other questions. – nbro Jan 17 at 18:34