I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).

Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...

Can anyone give an example of this?


1 Answer 1


Generally we can write for $R_c$ the total reward with added constant $c$ of a policy as $$ R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i $$ So if we have two policies with the same total reward (without added constant) $$ \sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i $$ but with different lengths $K_1 \neq K_2$ the total reward with added constant will be different, because the second term in $R_c$ ( $\sum_{i=0}^K c \gamma^i$ ) will be different.

As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as: $$ 10 + 0 + 0 + 0 = 10 $$ and $$ 0 + 10 = 10 $$ But when we add 100 to every reward: $$ 110 + 100 + 100 + 100 = 410 $$ and $$ 100 + 110 = 210 $$ Thus, now the first one is better.

In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R_c$ stays the same.

  • $\begingroup$ Excellent answer! $\endgroup$ Dec 14, 2020 at 22:13
  • $\begingroup$ Another important effect that distorts the problem and changes its solution is rewards sign changing. Explanation here at time 00:12 $\endgroup$ Mar 14, 2021 at 20:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .