What does $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ mean in the article Hindsight Experience Replay, section 2.1?

Question

Taken from section 2.1 in the article:

We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the environment is fully observable. An environment is described by a set of states $S$, a set of actions $A$, a distribution of initial states $p(s_0)$, a reward function $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, transition probabilities $p(s_{t+1} \mid s_t, a_t)$, and a discount factor $\gamma \in [0, 1]$.*

How should one interpret the maths behind it?

Answer 1

This answer assumes that you only have a problem with this notation from the article:

$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

This is a standard notation, used in many disciplines, for defining a function and its input and output domains. It is a bit like the method signature for the function - it does not fully define it, but does enough to show how it can interact with other expressions.

All functions can be thought of as maps between the input domain and output domain. You provide an input value, and it returns an output value. The values can be arbitrary mathematical objects. To show what kind of objects the inputs and outputs are allowed to be, the notation for sets is used.

Importantly the symbol $\mathbb{R}$ at the end does not refer to the set of possible rewards in the environment (although it is a reward function, and that will be its output), but the set of all real numbers, because a reward is always a real number*.

As a concrete example, if you had the function $f(x) = x^2 - 2x + 7$ defined for a real number $x$, then its equivalent notation might be $f : \mathbb{R} \rightarrow \mathbb{R}$. If you allowed $x$ to be complex then it would be $f : \mathbb{C} \rightarrow \mathbb{C}$, because $\mathbb{C}$ is the standard symbol for the set of all complex numbers.

So now we can break down the notation $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

$r$

The function is called $r$

$:$

It has an input domain of . . .

$\mathcal{S} \times \mathcal{A}$

The cartesian product of the set of all possible states $\mathcal{S}$ and the set of possible actions $\mathcal{A}$.

That is much the same as saying the function has a signature $r(s, a)$ where $s \in \mathcal{S}$ and $a \in \mathcal{A}$

$\rightarrow$

It has an output domain of . . .

$\mathbb{R}$

any single real number.

---

* This choice (of declaring the more general $\mathbb{R}$ instead of specific $\mathcal{R}$) is made partly because operators like $+$ and $\times$ are well defined for real numbers. This is a useful thing to assert about the behaviour of the reward function output when defining how value functions work for instance. Of course you could be more specific, defining $\mathcal{R}$ as some subset of $\mathbb{R}$, that would be correct and more precise definition, but it is not needed for general theory in reinforcement learning. The less precise definition is fine for nearly all purposes.