0
$\begingroup$

I noticed that there is no single "reward" convention in RL community and as a new RL learner this confuses me a lot.

For example, it is not clear to me if

Case 1: reward is a state property (shown next to a state, see example 1)

Case 2: reward is a state transition property (shown on the transition arcs, see example 2).

It is not clear also how you defined "returns" in both cases.

Ex1:

enter image description here

Ex2: enter image description here

Improve this question
$\endgroup$
5
  • $\begingroup$ @ nbro, Thank you. It answers partially: to see their differences. What left is, how do you define returns in each case? I mean how do you define returns separately for cases of using R(s), R(s,a) and R(s, a. a')? $\endgroup$
    – user3489173
    Feb 4, 2022 at 23:13
  • 1
    $\begingroup$ Returns are defined exactly the same in all cases - the discounted sum of rewards from any given timestep onwards. The form of the reward function, or its distribution doesn't change that at all. I.e. $G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+1+k}$ $\endgroup$
    – Neil Slater
    Feb 4, 2022 at 23:37
  • 1
    $\begingroup$ If you try to calculate the expectation of that return given a policy, the transition model, and one of the reward functions, then you might care a little about the form of the reward function, but only because you could move some forms of the function out of the sums that contain them for efficiency. $\endgroup$
    – Neil Slater
    Feb 4, 2022 at 23:39
  • $\begingroup$ @Neil Slater, what confuses me is this: for example, if you have a given sequence in ex1 (such as Class1-Class2-Class3-Pass-Sleep) and want to calculate the return associated to this sequence, you need the rewards associated to arcs (R(s, a ,s')) in my opinion, not the rewards associated to states (R(s)). But in David Silver's notes (this is where ex1 is taken), he calculates returns for a sequence from rewards associated to states (from R(s)). This confuses me. $\endgroup$
    – user3489173
    Feb 4, 2022 at 23:56
  • 1
    $\begingroup$ You can just associate the rewards to the arcs if you want, just copy all the rewards associated with states in ex1 to actions leading away from them (IIRC David Silver uses the reward values when leaving the associated states, i.e. they are considered to be rewards accrued from completing a time step in that state). It is functionally the same, because using $R(s)$ is a shortcut for stating $R(s,a,s')$ is the same for all $a,s'$ as per the duplicate question $\endgroup$
    – Neil Slater
    Feb 5, 2022 at 8:14

0

Reset to default

Browse other questions tagged .