1
$\begingroup$

Suppose, for simplicity sake, to be in a discrete time domain with the action set being the same for all states $S \in \mathcal{S}$. Thus, in a finite Markov Decision Process, the sets $\mathcal{A}$, $\mathcal{S}$, and $\mathcal{R}$ have a finite number of elements. We could then say the following

$$p(s',r | s,a) = P\{S_t=s',R_t=r | S_{t-1}=s,A_t=a\} ~~~ \forall s',s \in \mathcal{S}, r \in \mathcal{R} \subset \mathbb{R}, a \in \mathcal{A}$$

where the function $p$ defines the dynamics of the finite MDP and $P$ defines the probability.


How could I extend this to a general MDP? That is, an MDP where the sets $\mathcal{A}$, $\mathcal{S}$, and $\mathcal{R}$ haven't a finite number of elements? To be more precise, in my case $\mathcal{A} \subset \mathbb{R}^n$, $\mathcal{S} \subset \mathbb{R}^m$, and $\mathcal{R} \subset \mathbb{R}$. My thought is that the equation above is still true, however, the probability is zero for each tuple $s',r,s,a$.

Is it sufficient to say that for finite MDP we have

$$\sum_{s'\in\mathcal{S}}\sum_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$

while in non-finite MDP (supposing that the sets $\mathcal{s}$ and $\mathcal{A}$ are continuous) we have

$$\int_{s'\in\mathcal{S}}\int_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$

or is it more complex than this?

$\endgroup$
  • $\begingroup$ The tags didn't exist yet but I thought that they should. Feel free to modify them to MDP and finite-MDP if you think that would be better. $\endgroup$ – gvgramazio Nov 23 '18 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.