2
$\begingroup$

I am designing my own environment for a specific problem and I am thinking of the reward function for it. In some RL algos it is common to learn the reward that is associated with taking an action given the state. I therefore expect that a reward should always be the same when you encounter the exact same state and take the exact same action. Is this a constraint that really exists or is this a non-issue and the RL algorithm can deal with this variance in rewards received? I would like to center my rewards around 0.0 and it would be more intuitive to shape the rewards like this, but I am worried that because of the variance in rewards it might confuse the model.

$\endgroup$

2 Answers 2

3
$\begingroup$

The RL framework assumes only that the reward function depends on the current state, the selected action, and the next state:

$$\mathcal{R}(s_t, a_t, s_{t+1})$$

It can be deterministic but it can also be probabilistic. As long as it depends only on $(s_t, a_t, s_{t+1})$ its fine.

You can see equations (3.2) and (3.5) from the Sutton and Barto book.

$\endgroup$
1
  • 1
    $\begingroup$ May be worth saying, for clarity for the probabilistic result, that $\mathcal{R}(s_t, a_t, s_{t+1})$ returns the same distribution for the same arguments, where the deterministic version mentioned by OP would only return the same single value. Technically it doesn't even need to be the same distribution, just the same mean/expected immediate reward, but that would be an unusual MDP design. $\endgroup$ Sep 12, 2023 at 20:39
3
$\begingroup$

Maybe not clear from the other answer, but no, the reward depends usually also on the state you end up, which is not deterministic by definition of MDPs, and even in that case, the reward can be noisy.

Take the simplest case of Bandits, where you have a single state, and you can pull one of N levers, then when you pull a lever, you observe a reward that might change (like a lottery)

So, long story short, you can define any reward as a distribution, aka, no they are not deterministic

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .