2
$\begingroup$

The understanding I have is that they somehow adjust the objective to make it easier to meet, without changing the reward function.

... the observed proxy reward function is the approximate solution to a reward design problem

(source: Inverse Reward Design)

But I have trouble getting how they fit the overall reward objective and got confused by some examples of them. I had the idea of them being small reward functions (as in the case of solving for sparse rewards) eventually leading to the main goal. But the statement below, from this post, made me question that.

Typical examples of proxy reward functions include “partial credit” for behaviors that look promising; artificially high discount rates and careful reward shaping;...

  1. What are they, and how would one go about identifying and integrating proxy rewards in an RL problem?

  2. In the examples above, how would high discount rates form a proxy reward?

I'm also curious about how they are used as a source of multiple rewards

Improve this question
$\endgroup$
6
  • $\begingroup$ Maybe have a look at ai.stackexchange.com/q/20040/2444, although I think that you know what "proxy" usually means. $\endgroup$ – nbro♦ Sep 22 '20 at 11:24
  • $\begingroup$ So "proxies" are intermediaries. This helped add clarity into the "what are they" part of the question $\endgroup$ – mugoh Sep 23 '20 at 5:05
  • $\begingroup$ Please, read my answer below ai.stackexchange.com/a/24074/2444, if you haven't yet. You should accept it, if it answers your question. I read that IRD paper and other related papers, so I'm pretty confident that the answer is correct. If something is still unclear, let me know. Please, take a look at ai.stackexchange.com/help/someone-answers for more details. $\endgroup$ – nbro♦ Jan 16 at 18:17
  • $\begingroup$ Thanks for pointing this out @nbro. I was also seeking clarity on how high discount rates form proxy rewards and hiw proxies can be used as a source of multiple rewards $\endgroup$ – mugoh Jan 17 at 4:13
  • $\begingroup$ It's not indeed clear what the article's author meant by "artificially high discount rates". It's not even clear if they are referring to the discount factor of the RL algorithm. Even if that was the case, it's not very clear why that would be a "proxy reward function". The discount factor is the way to weight short-term and long-term rewards, but it's not very clear why one would consider that a "proxy reward function". Maybe this has to do with the "long time horizon" part of that article. Maybe the best thing to do is e-mail the article's author and ask him to answer your other questions. $\endgroup$ – nbro♦ Jan 17 at 18:34
1
$\begingroup$

In the paper that you cite, Inverse Reward Design (2017), the authors actually define what they mean by "proxy reward function".

We formalize this in a probabilistic model that relates the proxy (designed) reward to the true reward

So, the proxy reward function is the reward function designed by the human, which may not necessarily be the reward function that he/she intended (i.e. it may be a misspecified reward function), given that the human may have forgotten to model/incorporate certain (unpredicted by the human) scenarios or situations that the agent may face. This usage of the word "proxy" is thus consistent with the general usage of the word in computer science, i.e. a "proxy reward function" is a reward function that is used instead of the intended (optimal) reward function.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.