Can a reward function have various cases?

Question

I'm doing a Q-learning algorithm and I'm designing my reward function. Basically I'm working on optimizing a network while changing some parameters. My metric to measure its optimization is the delay on a flow of data (generated).

I've discretized my delay values in intervals and now I'm designing the rewards. What I was thinking about is, as I want to prioritize lower delays to use a non linear function such as $1/x^2$. The range of my delays are usually from $10$ to $40$ secs (depending on the perturbations). It works well when the delay is varying a lot, but way less good when the delay isn't varying that much (with low perturbations).

I was then wondering if there are restrictions on reward functions. What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under $15$ secs can I normalize the value? And use another way to calculate reward when it's in an interval above?

I'm new to reinforcement learning so maybe what I said is non sense, but I would gladly hear any advice or idea.

Adrien

Answer 1

In general the reward function can be anything that allows you to solve the task: if the reward is misspecified, the env cannot be solved; this is the reward hypothesis which RL relies on.

So, consider the reward function to be defined as $r(s,a)$ or even $r(s,a,s')$ from a state $s$, actions $a$, and sometimes including also the next state $s'$. It can be as much complex as you want, having various cases, parts, and so on. But consider that if too complex you may find a hard time debugging it, to see if the reward fn suits your task. Also, different rewards (assuming they are correct enabling to solve the problem) may result in slower or faster convergence to optimality.

What I wanted to do is to have different parts depending on the value. Like if my value is in an interval under 15 secs can I normalize the value? And use another way to calculate reward when it's in an interval above?

In principle you can do that, as I said above. Try to have a reward range that is bounded and possibly limited (e.g. $[-1, 1]$), avoiding large values and occasional spikes.

For example, you mentioned that $1/x^2$ works well when the delays are quite different. In the case the delays are similar, you can try to transform them (e.g. by $\exp$ to increase the difference) and then apply the above function. For example, case 1 is $1/x^2$, case 2 can be $1/\exp(x)$, case 3 something else, etc..