Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

Question

I'm taking a Coursera course on Reinforcement learning. There was a question there that wasn't addressed in the learning material: Does adding a constant to all rewards change the set of optimal policies in episodic tasks?

The answer is Yes - Adding a constant to the reward signal can make longer episodes more or less advantageous (depending on whether the constant is positive or negative).

Can anyone explain why is this so? And why it doesn't change in the case of continuous (non episodic) tasks? I don't see why adding a constant matters - as an optimal policy would still want to get the maximum reward...

Can anyone give an example of this?

Answer 1

Generally we can write for $R_c$ the total reward with added constant $c$ of a policy as $$ R_c = \sum_{i=0}^K (r_i + c) \gamma^i = \sum_{i=0}^K r_i \gamma^i + \sum_{i=0}^K c \gamma^i $$ So if we have two policies with the same total reward (without added constant) $$ \sum_{i=0}^{K_1} r_i^1 \gamma^i = \sum_{i=0}^{K_2} r_i^2 \gamma^i $$ but with different lengths $K_1 \neq K_2$ the total reward with added constant will be different, because the second term in $R_c$ ( $\sum_{i=0}^K c \gamma^i$ ) will be different.

As an example: Consider two optimal policies, both generating the same cumulative reward of 10, but the first policy visits 4 states, before it reaches a terminal state, while the second visits only two states. The rewards can be written as: $$ 10 + 0 + 0 + 0 = 10 $$ and $$ 0 + 10 = 10 $$ But when we add 100 to every reward: $$ 110 + 100 + 100 + 100 = 410 $$ and $$ 100 + 110 = 210 $$ Thus, now the first one is better.

In the continious case, the episodes always have length $K = \infty$. Therefore, they always have the same length, and adding a constant doesnt change anything, because the second term in $R_c$ stays the same.