How to normalizing various elements of the reward function?

Question

Suppose I have a reward function $R$ that I wish to penalize w.r.t two distinct phenomenons $A$ and $B$. $A$, for example, could represent the phenomenon of the state not crossing some boundary $[s_1,s_2]$ and $B$ can represent the phenomenon that two consecutive actions shouldn't be too far apart $|a_t - a_{t+1}| < \epsilon$, for some small $\epsilon$. A trivial reward mechanism can be:

step(prev_a, a,epsilon, min_s, max_s):
    s = env(action)
    if not min_s < s < max_s: 
        r -= s
    if ||a-prev_a|| > epsilon:
        r -= a

As $A$ and $B$ are from different worlds (different physical units, if you will), they both have different ranges. For example, a state $s$ may obtain values that are at most $10$, though an action $a$ may have larger values like $100$. Hence, penalizing by subtracting the state or the action from the current reward may lead to the preference of the agent to only make sure the action condition is set, as this one translates to more future rewards.

How can this issue be addressed? I assume some normalization should be added, though I'm not quite sure how.

Any ideas?

Answer 1

There are several ways to normalize various elements of the reward function. One way would be to divide all values by the largest value in the set. This would ensure that all values are between $0$ and $1$. Another way would be to subtract the mean from all values and then divide it by the standard deviation. This would ensure that all values have a mean of $0$ and a standard deviation of $1$.

A trivial reward function for this scenario would be to simply assign a negative value to $R$ whenever either $A$ or $B$ occurs. However, this may not be the most effective way to penalize the behavior. A more effective approach might be to weigh the penalties differ depending on how severe the violation is. For example, if $A$ occurs, the penalty could be $-1$, but if $B$ occurs, the penalty could be $-10$. This would ensure that more severe violations are penalized more heavily than less severe violations.

This reward function appears to be penalizing the agent for taking actions that result in the state not being within some boundary, as well as for taking actions that are too far apart from the previous action.

One way to avoid this issue would be to normalize the values of $A$ and $B$ before subtracting them from the current reward. This would ensure that both $A$ and $B$ have an equal impact on the reward function, regardless of their range.