12
$\begingroup$

On Sutton and Barto's RL book, the reward hypothesis is stated as

that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

Are there examples of tasks where the goals and purposes cannot be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal?

All I can think of are tasks with subjective rewards, like "writing good music", but I am not convinced because maybe this is actually definable (perhaps by some super-intelligent alien) and we just aren't smart enough yet. Thus, I'm especially interested in counterexamples that logically or provably fail the hypothesis.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Well, "all of what we mean by goals and purposes" has some wriggle room which is open to interpretation and stretching the meanings of things. Subjective rewards are easy problem to fix, you just need to go to the source of subjective measurement - for example just ask a large enough cross-selection of people whether the output was "good music" and take an average. Or even just count downloads of the latest automated hit single. That is hard to do in practice, and slow perhaps, but the hypothesis says nothing about how convenient the reward signal needs to be. $\endgroup$
    – Neil Slater
    Nov 19, 2019 at 10:18
  • $\begingroup$ @Bananin Are you looking for specific examples in the context of reinforcement learning or any example we can think of? $\endgroup$
    – nbro
    Nov 19, 2019 at 15:14
  • $\begingroup$ @Bananin you mention subjective rewards are counter examples but they are not, theyre just difficult to model. Like Neil said, you have to find the source of disambiguity: in his examples he offers a commonly used heuristic which is to find the expectation over some set of subjective representatives to define a more "objective" measure-- but regardless subjectivity can be made objective by conditioning on the subjective factor "good vs what I believe to be good" in which case you can still use the above concept. $\endgroup$
    – mshlis
    Nov 19, 2019 at 16:58
  • $\begingroup$ By the way, the reward hypothesis was suggested by Michael Littman, according to Barto & Sutton's book (2nd edition, p. 70). $\endgroup$
    – nbro
    Nov 18, 2020 at 21:52

4 Answers 4

Reset to default
4
$\begingroup$

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I like your perspective! However, I think the issue wouldn't be the different prioritization across agents (the reward hypothesis would fit them individually), but the impossibility of such prioritization. Example: COVID lockdowns. They keep people from getting infected but hurt the economy. We want to optimize for economy and health, but how do we add them up? $\endgroup$
    – Bananin
    Aug 19, 2020 at 0:44
  • 1
    $\begingroup$ it being /unclear/ as to how it will collapse into a single dimension value itself does not nessicate that there isn't one. or, perhaps, one can imagine there being a very complex function that considers pretty much everything imaginable into account $\endgroup$
    – k.c. sayz 'k.c sayz'
    Aug 19, 2020 at 6:21
  • 1
    $\begingroup$ What you are describing is addressed in a field of research known as multi-objective reinforcement learning (MORL), which uses vector rewards. I'd suggest you read the following survey paper for an overview of MORL (disclaimer - I was a co-author on this, but I genuinely think it is a useful introduction to this area) Roijers, D. M., Vamplew, P., Whiteson, S., & Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research, 48, 67-113. jair.org/index.php/jair/article/view $\endgroup$
    – Peter Vamplew
    Jan 13, 2021 at 4:06
  • 1
    $\begingroup$ I don't think that this is a counterexample to the reward hypothesis. It doesn't matter whether or not we know the optimal way of combining the objectives to achieve the optimal multi-objective or whether the combination of multiple objectives to form the scalar-based one leads to our desired behaviour. The RH just states that we can maximize a scalar-based objective to achieve some goal, but it says nothing about how to design that objective. $\endgroup$
    – nbro
    Jul 26, 2021 at 12:17
2
$\begingroup$

I believe that there is no clear answer to your question. It essentially boils down to whether you are a reductionist – whether you believe that quantitative measurements can truly give justice to the complexity of the real world, and that a framework such as expectation maximization can losslessly capture what we care about as humans in the performing of tasks.

From a non-reductionist perspective, one would be aware that almost any mathematical representation of complex real-world goals will necessarily be a proxy rather than the true goal (as many goals are not mathematically formalizable, such as what we perceive as "good music" or "meaning"), and thus the reward hypothesis is at best an approximation. Based on this, a non-reductionist's reward hypothesis could be rephrased as:

that all of what we mean by goals and purposes can be well thought of approximately operationalized (albeit at a certain domain-dependent loss) as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

Clearly the original (stricter) version of the reward hypothesis does apply to some cases, such as purely-quantitative domains (e.g. maximizing $ earned on the stock market, or maximizing score in a video game), but as soon as the problem involves enough "complexity" (e.g. humans, or wherever you think the boundary should be), a non-reductionist would say that mathematics is clearly not fit to the task to truly capture the intended goal.

More info on the reward hypothesis (as presented by Michael Littman himself) is here. I would have added it as a comment to the question but do not have enough reputation.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I think there's a typo in "Clearly this doesn't apply to all cases, such as purely-quantitative domains", i.e. why wouldn't it clearly apply to "purely-quantitative"? $\endgroup$
    – nbro
    Dec 2, 2020 at 23:51
  • $\begingroup$ It's actually not a typo – what I meant is that if your goal is purely defined within a quantitative system (the score in a game, the number of dollars in your bank account), then a scalar reward might in fact not be a loss-full reduction of your goal, but your goal itself. However, if one investigates even these cases closely, usually these goals turn out to also be simplifications of more fuzzy goals ("maximize the $ in my bank account while being ethical"). $\endgroup$
    – mdc
    Dec 3, 2020 at 17:35
  • 1
    $\begingroup$ Maybe you didn't get my point or I don't fully get what you wrote (though I think I do). You're saying that the reward hypothesis (RH) doesn't apply to all cases, such as purely quantitative domains, for example, maximizing the score of a game, but that is one of the examples where the RH usually applies (assuming that the goal is not fuzzy). $\endgroup$
    – nbro
    Dec 3, 2020 at 17:48
  • $\begingroup$ Thanks for the comment – edited my original post for clarity $\endgroup$
    – mdc
    Dec 6, 2020 at 21:51
0
$\begingroup$

The closest counterexamples I can think of are cases where reward shaping is required to learn a good policy but ends up having unintended consequences.

Reward shaping is usually used in cases we want to encourage a particular behavior or when the reward is sparse or when capturing exactly what you want is not straightforward or infeasible. But it is not a good practice to rely too much on it as it can have unintended consequences. A simple example of this is described here https://openai.com/blog/faulty-reward-functions/.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I don't understand how reward shaping contradicts the reward hypothesis. Can you clarify that? $\endgroup$
    – nbro
    Jul 1, 2020 at 16:37
  • $\begingroup$ I will try directly to answer the question you asked here. As you said in the comment any task can be modeled with rewards with some assumptions. But whether the modeling leads to the intended original task getting solved is not guaranteed. I consider this as a failure of reward modeling and hence these tasks might not fall under the reward maximization assumption. $\endgroup$
    – saiRegrefree
    Jul 2, 2020 at 18:06
  • $\begingroup$ When I think about it, my point is actually targeting the cases where the original reward modeling doesn't work well with the existing training techniques (because of sparse reward or reward shaping). You can argue both ways I think. $\endgroup$
    – saiRegrefree
    Jul 2, 2020 at 18:09
  • $\begingroup$ The reward hypothesis says that a goal can be thought of as the maximization of the reward. Either if you find a maximum or not of the objective function doesn't imply that you cannot model a problem as the maximization of the reward. So, I really don't think that reward shaping goes against the reward hypothesis. $\endgroup$
    – nbro
    Jul 2, 2020 at 18:18
-2
$\begingroup$

The book sets this hypothesis up by laying out a few assumptions:

In reinforcement learning, the purpose or goal of the agent is formulated in terms of a special signal called the reward, passing from the environment to the agent. At each time step, the reward is a simple number.

We could think about what counterexamples to those assumptions might be:

  1. The reward signal originates internally, instead of originating from the environment. (e.g. meditation, or abstract introspection)
  2. The signal is not received every time step, or isn't necessarily expected to be received at all. (e.g. seeking of transcendent experiences)

What might be common for these counterexamples is that the reinforcement learning mechanism itself undergoes spontaneous change. A signal that would have been positive before the spontaneous change might now be negative. The reward landscape itself might be completely different. From the agent's perspective, it might be impossible to evaluate what changed. The agent might have a 'subconscious' secondary algorithm that introduces changes in the learning algorithm itself, in a way that's decoupled from any reward-defined behavior.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Why would point 1 contradict the reward hypothesis? It should not matter where the reward originates from. Your second point seems to be about sparse rewards, but that still doesn't contradict the reward hypothesis (I think). $\endgroup$
    – nbro
    Jul 1, 2020 at 16:40
  • $\begingroup$ It seems like @Bananin was looking for some examples that are hard to logically navigate through in the way traditional reward accumulation is. Both points come down to an inability to define or accumulate rewards. If rewards are spontaneous, or if the agent changes its accumulation mechanism between rewards, it makes things difficult to extrapolate and model. $\endgroup$
    – bey
    Jul 4, 2020 at 18:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .