4
$\begingroup$

enter image description here

The above picture is from a textbook on RL. I am confused with the notation $r_t=\mathcal{R}(s_t, a_t, s_{t+1})$. How can reward at time step $t$ be a function of a state at time step $t+1$?

$\endgroup$
1

3 Answers 3

2
$\begingroup$

There is a free choice when deciding the indexing of rewards in reinforcement learning. You can decide to associate the immediate reward with the time step of state and action that generated it, or with the time step of the observed next state. Both conventions are self-consistent and unambiguous, they differ only in the notation used in expressions.

The most common convention is to associate reward with next time step, leading to trajectories that look like this:

$$s_0, a_0, r_1, s_1, a_1, r_2 . . . s_{T-1}, a_{T-1}, r_T, s_T$$

The alternative convention, that it looks like your excerpt is using*, looks like this:

$$s_0, a_0, r_0, s_1, a_1, r_1 . . . s_{T-1}, a_{T-1}, r_{T-1}, s_T$$

Whether you use 0 as first index or 1 is a similar convention.

In practice, the next state and reward that result from a state/action pair are part of an environment response where they can be calculated in any order. In some cases, $s_{t+1}$ is required before the associated reward, because the value of the reward depends on it, yet usually reward is shown in trajectories before the next state so that the listing has the same pattern $s,a,r$ throughout. It doesn't matter how the notation tracks that, as long as it is consistent. The timing of events is not different in the different conventions, only the index numbering.


* Note that the diagram that goes with the equations does show reward and state sharing time step indexing. This implies that there could be a typo and the author meant to write $r_{t+1}$ where they have put $r_t$ in equation 1.3. Or perhaps they have used two sources which use different conventions and not noticed.

$\endgroup$
2
  • $\begingroup$ So, you also agree that the equations and the diagram are not consistent? That was in fact my question. $\endgroup$ Feb 1 at 2:53
  • $\begingroup$ @DSPinfinity yes I agree, something looks wrong there. However in isolation neither is incorrect $\endgroup$ Feb 1 at 8:16
-1
$\begingroup$

I don't know that I've seen that notation but one must remember that an MDP does not guarantee a transition from $S_1$ to $S_2$ via action $A_0$. There may be some stochasity in the transition causing the agent to stay in $S_1$, successfully transition to $S_2$ or instead go to $S_3$. Think of a grid world with a slippery floor that may cause the agent not to move or to over move.

The reward is not for taking action $A_0$ in $S_1$ but for the real resulting state. Therefore the reward at time t is always for $S_{t+1}$.

$\endgroup$
-1
$\begingroup$

Your question didn't specify the reference of your picture at all which may have additional context in terms of the specific meaning of your reward function such as whether it's a random variable or an expected value. For instance, your picture very much resembles Figure 3.1 of Sutton's RL book where the reward $\mathcal{R}$ is a random variable and they never wrote or needed $\mathcal{R}(s_t, a_t, s_{t+1})$, but instead, its expected value $r(s_t, a_t, s_{t+1})$ is defined on page 49 due to possible stochastic environment unlike a normal grid world game:

We can also compute the expected rewards for state–action pairs as a two-argument function $r : \mathcal{S} \times \mathcal{A} \to \mathbb{R}$: $$r(s,a) = \sum_{r \in \mathcal{R}} \sum_{s' \in \mathcal{S}} rp(s',r|s,a)$$ and the expected rewards for state–action–next-state triples as a three-argument function $r : \mathcal{S} \times \mathcal{A} \times \mathcal{S} \to \mathbb{R}$, $$r(s,a,s') = \sum_{r \in \mathcal{R}}r\frac{p(s',r|s,a)}{p(s'|s,a)}$$

In MDP RL there's always supposed to be an explicit model of the environment including stationary state and reward transition dynamics, thus as you claimed there's no necessary need for the (expected) reward function given by the environment to ever have explicit dependence on the next entering state $s_{t+1}$ to solve any problem. In fact in Sutton's book the only place the 3-argument expected reward is explicitly used appears in a nondeterministic finite state diagram for a recycling robot example on page 52:

enter image description here

You see in such a diagram the expected reward function needs the explicit dependence of the entering state to conveniently express its value based on its entering state for each of the possibly branching state transition due to its random environment characterized by the above Markovian transition dynamics $p(s',r|s,a)$, and they're obviously mathematically compatible with the above usual 2-argument reward function.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .