0
$\begingroup$

Taken from section 2.1 in the article:

We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume that the environment is fully observable. An environment is described by a set of states $S$, a set of actions $A$, a distribution of initial states $p(s_0)$, a reward function $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$, transition probabilities $p(s_{t+1} \mid s_t, a_t)$, and a discount factor $\gamma \in [0, 1]$.*

How should one interpret the maths behind it?

$\endgroup$
  • 1
    $\begingroup$ Are you only asking about the notation $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ and want clarification on that bit? It is not clear. The "How should one interpret the maths behind it?" is not clear, because you don't explain which parts you do not understand, and the article seems to be literally the maths explained concisely. It looks like you need some of the maths broken down into more descriptive sentences and multiple steps, but which parts? Could you also please link the original article. Use edit to provide details. $\endgroup$ – Neil Slater Sep 14 at 8:21
  • $\begingroup$ Hello. Welcome to AI SE! Note that you can use latex on this site. I've edited your post to use it, but please in the next post use latex. And don't forget to clarify your question, as the previous comment suggests. $\endgroup$ – nbro Sep 14 at 10:18
3
$\begingroup$

This answer assumes that you only have a problem with this notation from the article:

$r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

This is a standard notation, used in many disciplines, for defining a function and its input and output domains. It is a bit like the method signature for the function - it does not fully define it, but does enough to show how it can interact with other expressions.

All functions can be thought of as maps between the input domain and output domain. You provide an input value, and it returns an output value. The values can be arbitrary mathematical objects. To show what kind of objects the inputs and outputs are allowed to be, the notation for sets is used.

Importantly the symbol $\mathbb{R}$ at the end does not refer to the set of possible rewards in the environment (although it is a reward function, and that will be its output), but the set of all real numbers, because a reward is always a real number*.

As a concrete example, if you had the function $f(x) = x^2 - 2x + 7$ defined for a real number $x$, then its equivalent notation might be $f : \mathbb{R} \rightarrow \mathbb{R}$. If you allowed $x$ to be complex then it would be $f : \mathbb{C} \rightarrow \mathbb{C}$, because $\mathbb{C}$ is the standard symbol for the set of all complex numbers.

So now we can break down the notation $r : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$

$r$

The function is called $r$

$:$

It has an input domain of . . .

$\mathcal{S} \times \mathcal{A}$

The cartesian product of the set of all possible states $\mathcal{S}$ and the set of possible actions $\mathcal{A}$.

That is much the same as saying the function has a signature $r(s, a)$ where $s \in \mathcal{S}$ and $a \in \mathcal{A}$

$\rightarrow$

It has an output domain of . . .

$\mathbb{R}$

any single real number.


* This choice (of declaring the more general $\mathbb{R}$ instead of specific $\mathcal{R}$) is made partly because operators like $+$ and $\times$ are well defined for real numbers. This is a useful thing to assert about the behaviour of the reward function output when defining how value functions work for instance. Of course you could be more specific, defining $\mathcal{R}$ as some subset of $\mathbb{R}$, that would be correct and more precise definition, but it is not needed for general theory in reinforcement learning. The less precise definition is fine for nearly all purposes.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.