# Does $R_{s}=E[R_{t}|S_{t}=s]$ indicate the reward we might expect on getting on average moving from any other state to $s$?

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $$R_{s}$$ the reward function. I used to think that it's the reward we may expect on average after taking an action $$A_{t}$$ at state $$S_{t}=s$$ and at time step $$s$$ but thinking about it more I think I was wrong.

If we consider that the agent at time step $$t$$ receives from the environment both an observation $$O_{t}$$ and a reward $$R_{t}$$ which will make up the agent's state $$S_{t}=s$$ depending on what function of the history we take, then it means that we got to state $$S_{t}=s$$ from a state $$S_{t-1}$$ taking action an action $$A_{t-1}$$ right?

I'd like to ask you to confirm or not my claim.

$$R_{s}=\mathbb{E}[R_{t}|S_{t}=s]$$

is the expected reward at time step $$t$$ given that the state at time $$t$$ is $$s$$, where

• $$R_{t}$$ and $$S_t$$ are random variables that represent the reward and state at time $$t$$, respectively,
• $$S_{t}=s$$ is an event, and
• $$\mathbb{E}[X]$$ denotes the expected value of the random variable $$X$$.

It does not matter how you entered $$S_t = s$$. The only thing that matters is that the current state is $$s$$. So, the answer to your question in the title is - no. $$R_s$$ is defined as an expectation (average) of the reward for being in a state, and, in this case, it doesn't take into account next states.

Suppose that the reward space $$\mathcal{R} \subset \mathbb{R}$$ is a discrete set, then we can write $$R_s$$ as follows

$$R_{s}=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s),$$

where $$p(r \mid S_{t}=s)$$ is a conditional probability distribution that describes how the reward is distributed in a given state. If you always get the same reward, i.e. $$p(r \mid S_{t}=s) = 1$$ for a particular $$r$$ and $$0$$ for all others, then you have a deterministic reward function, i.e. $$R_s$$ is equal to the $$r$$ for which $$p(r \mid S_{t}=s) = 1$$. In most RL examples, reward functions are deterministic.

Note that it is probably more common to define the reward function as a function of the state $$S_t = s$$ and action taken in $$s$$, $$A_t =a$$, and write it as $$R(s, a)$$ or $$r(s, a)$$, i.e.

\begin{align} R(s, a) &=\mathbb{E}[R_{t}|S_{t}=s, A_t = a]\\ &=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s, A_t = a) \end{align}

Note that we're now using $$p(r \mid S_{t}=s, A_t = a)$$ and not $$p(r \mid S_{t}=s)$$.

You could also define $$R(s, a, s')$$. See this or Sutton & Barto's book for more details.

• I see, thank you for clarifying this and for providing the link. Apr 17 at 1:05