0
$\begingroup$

I'm trying to understand correctly what each "variable" in RL is and I'm not sure about $R_{s}$ the reward function. I used to think that it's the reward we may expect on average after taking an action $A_{t}$ at state $S_{t}=s$ and at time step $s$ but thinking about it more I think I was wrong.

If we consider that the agent at time step $t$ receives from the environment both an observation $O_{t}$ and a reward $R_{t}$ which will make up the agent's state $S_{t}=s$ depending on what function of the history we take, then it means that we got to state $S_{t}=s$ from a state $S_{t-1}$ taking action an action $A_{t-1}$ right?

I'd like to ask you to confirm or not my claim.

$\endgroup$

1 Answer 1

1
$\begingroup$

$$R_{s}=\mathbb{E}[R_{t}|S_{t}=s]$$

is the expected reward at time step $t$ given that the state at time $t$ is $s$, where

  • $R_{t}$ and $S_t$ are random variables that represent the reward and state at time $t$, respectively,
  • $S_{t}=s$ is an event, and
  • $\mathbb{E}[X]$ denotes the expected value of the random variable $X$.

It does not matter how you entered $S_t = s$. The only thing that matters is that the current state is $s$. So, the answer to your question in the title is - no. $R_s$ is defined as an expectation (average) of the reward for being in a state, and, in this case, it doesn't take into account next states.

Suppose that the reward space $\mathcal{R} \subset \mathbb{R}$ is a discrete set, then we can write $R_s$ as follows

$$R_{s}=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s),$$

where $p(r \mid S_{t}=s)$ is a conditional probability distribution that describes how the reward is distributed in a given state. If you always get the same reward, i.e. $p(r \mid S_{t}=s) = 1$ for a particular $r$ and $0$ for all others, then you have a deterministic reward function, i.e. $R_s$ is equal to the $r$ for which $p(r \mid S_{t}=s) = 1$. In most RL examples, reward functions are deterministic.

Note that it is probably more common to define the reward function as a function of the state $S_t = s$ and action taken in $s$, $A_t =a$, and write it as $R(s, a)$ or $r(s, a)$, i.e.

\begin{align} R(s, a) &=\mathbb{E}[R_{t}|S_{t}=s, A_t = a]\\ &=\sum_{r \in \mathcal{R}} r p(r \mid S_{t}=s, A_t = a) \end{align}

Note that we're now using $p(r \mid S_{t}=s, A_t = a)$ and not $p(r \mid S_{t}=s)$.

You could also define $R(s, a, s')$. See this or Sutton & Barto's book for more details.

$\endgroup$
1
  • 1
    $\begingroup$ I see, thank you for clarifying this and for providing the link. $\endgroup$
    – Daviiid
    Apr 17 at 1:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .