On Sutton and Barto's RL book, the reward hypothesis is stated as

that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward)

Are there examples of tasks where the goals and purposes cannot be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal?

All I can think of are tasks with subjective rewards, like "writing good music", but I am not convinced because maybe this is actually definable (perhaps by some super-intelligent alien) and we just aren't smart enough yet. Thus, I'm especially interested in counterexamples that logically or provably fail the hypothesis.

  • 2
    $\begingroup$ Well, "all of what we mean by goals and purposes" has some wriggle room which is open to interpretation and stretching the meanings of things. Subjective rewards are easy problem to fix, you just need to go to the source of subjective measurement - for example just ask a large enough cross-selection of people whether the output was "good music" and take an average. Or even just count downloads of the latest automated hit single. That is hard to do in practice, and slow perhaps, but the hypothesis says nothing about how convenient the reward signal needs to be. $\endgroup$ – Neil Slater Nov 19 '19 at 10:18
  • $\begingroup$ @Bananin Are you looking for specific examples in the context of reinforcement learning or any example we can think of? $\endgroup$ – nbro Nov 19 '19 at 15:14
  • $\begingroup$ @Bananin you mention subjective rewards are counter examples but they are not, theyre just difficult to model. Like Neil said, you have to find the source of disambiguity: in his examples he offers a commonly used heuristic which is to find the expectation over some set of subjective representatives to define a more "objective" measure-- but regardless subjectivity can be made objective by conditioning on the subjective factor "good vs what I believe to be good" in which case you can still use the above concept. $\endgroup$ – mshlis Nov 19 '19 at 16:58

What if a scalar reward is insufficient, or its unclear on how to collapse a multi-dimensional reward to a single dimension. Example, for someone eating a burger, both taste and cost are important. Agents may prioritize taste and cost differently, so its not clear on how to aggregate the two. It is also not clear on how a subjective categorical taste value can be combined with a numerical cost.

| improve this answer | |
  • $\begingroup$ I like your perspective! However, I think the issue wouldn't be the different prioritization across agents (the reward hypothesis would fit them individually), but the impossibility of such prioritization. Example: COVID lockdowns. They keep people from getting infected but hurt the economy. We want to optimize for economy and health, but how do we add them up? $\endgroup$ – Bananin Aug 19 at 0:44
  • $\begingroup$ it being /unclear/ as to how it will collapse into a single dimension value itself does not nessicate that there isn't one. or, perhaps, one can imagine there being a very complex function that considers pretty much everything imaginable into account $\endgroup$ – k.c. sayz 'k.c sayz' Aug 19 at 6:21

The closest counterexamples I can think of are cases where reward shaping is required to learn a good policy but ends up having unintended consequences.

Reward shaping is usually used in cases we want to encourage a particular behavior or when the reward is sparse or when capturing exactly what you want is not straightforward or infeasible. But it is not a good practice to rely too much on it as it can have unintended consequences. A simple example of this is described here https://openai.com/blog/faulty-reward-functions/.

| improve this answer | |
  • 1
    $\begingroup$ I don't understand how reward shaping contradicts the reward hypothesis. Can you clarify that? $\endgroup$ – nbro Jul 1 at 16:37
  • $\begingroup$ I will try directly to answer the question you asked here. As you said in the comment any task can be modeled with rewards with some assumptions. But whether the modeling leads to the intended original task getting solved is not guaranteed. I consider this as a failure of reward modeling and hence these tasks might not fall under the reward maximization assumption. $\endgroup$ – saiRegrefree Jul 2 at 18:06
  • $\begingroup$ When I think about it, my point is actually targeting the cases where the original reward modeling doesn't work well with the existing training techniques (because of sparse reward or reward shaping). You can argue both ways I think. $\endgroup$ – saiRegrefree Jul 2 at 18:09
  • $\begingroup$ The reward hypothesis says that a goal can be thought of as the maximization of the reward. Either if you find a maximum or not of the objective function doesn't imply that you cannot model a problem as the maximization of the reward. So, I really don't think that reward shaping goes against the reward hypothesis. $\endgroup$ – nbro Jul 2 at 18:18

The book sets this hypothesis up by laying out a few assumptions:

In reinforcement learning, the purpose or goal of the agent is formulated in terms of a special signal called the reward, passing from the environment to the agent. At each time step, the reward is a simple number.

We could think about what counterexamples to those assumptions might be:

  1. The reward signal originates internally, instead of originating from the environment. (e.g. meditation, or abstract introspection)
  2. The signal is not received every time step, or isn't necessarily expected to be received at all. (e.g. seeking of transcendent experiences)

What might be common for these counterexamples is that the reinforcement learning mechanism itself undergoes spontaneous change. A signal that would have been positive before the spontaneous change might now be negative. The reward landscape itself might be completely different. From the agent's perspective, it might be impossible to evaluate what changed. The agent might have a 'subconscious' secondary algorithm that introduces changes in the learning algorithm itself, in a way that's decoupled from any reward-defined behavior.

| improve this answer | |
  • 1
    $\begingroup$ Why would point 1 contradict the reward hypothesis? It should not matter where the reward originates from. Your second point seems to be about sparse rewards, but that still doesn't contradict the reward hypothesis (I think). $\endgroup$ – nbro Jul 1 at 16:40
  • $\begingroup$ It seems like @Bananin was looking for some examples that are hard to logically navigate through in the way traditional reward accumulation is. Both points come down to an inability to define or accumulate rewards. If rewards are spontaneous, or if the agent changes its accumulation mechanism between rewards, it makes things difficult to extrapolate and model. $\endgroup$ – bey Jul 4 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.