I have two signals that I want to use to model my reward.

The first one is the CPU TIME: running mean from this diagram:

enter image description here

The second one is the MAX RESIDUAL from this diagram:

enter image description here

Since they are both equally important, I can weight them together like this:

$r = w_\rho \rho + w_\tau \tau$

where $r$ is the reward function, $\tau$ is the CPU TIME: running mean, and $\rho$ is the MAX RESIDUAL. The problem is, how to set the weights $w_\tau,w_\rho$ to make the contributions equally important if $\rho$ and $\tau$ are on very different scales?

RL algorithms will learn policies based on increases/decreases of the reward, and if one signal has values that are much smaller than the other, it will influence the reward less, if this is done in a wrong way.

On the other hand, if the algorithm converges, they must be on different scales, as I want the CPU time to go ideally to $0$, and residuals $\rho$ to be minimized as well.

Modeling the reward function is a crucial RL step, because it decides what the algorithm will in fact optimize. How are examples like these handled? Are there any best practices for this? Also, what happens when there are $n$ such signals, that have to be combined with "equally important" weighting into a reward function?

Basing the weights $w$ on the current signal values is possible to define their reward, but then the reward contributions won't see $\max(\rho), \min(\rho)$ and $\max(\tau), \min(\tau)$ over time.

So, how do you do feature scaling for reward signals?

  • 1
    $\begingroup$ scale your rewards to be 0 mean and variance 1? $\endgroup$ – David Ireland Aug 7 at 12:52
  • $\begingroup$ @DavidIreland I tried z-scaling already and it doesn't seem to scale to same min,max intervals for "running" signals... can you suggest an alternative scaling for running signals that does what you suggest? Thanks, btw :) $\endgroup$ – tmaric Aug 7 at 13:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.