# How to generalize finite MDP to general MDP?

Suppose, for simplicity sake, to be in a discrete time domain with the action set being the same for all states $$S \in \mathcal{S}$$. Thus, in a finite Markov Decision Process, the sets $$\mathcal{A}$$, $$\mathcal{S}$$, and $$\mathcal{R}$$ have a finite number of elements. We could then say the following

$$p(s',r | s,a) = P\{S_t=s',R_t=r | S_{t-1}=s,A_t=a\} ~~~ \forall s',s \in \mathcal{S}, r \in \mathcal{R} \subset \mathbb{R}, a \in \mathcal{A}$$

where the function $$p$$ defines the dynamics of the finite MDP and $$P$$ defines the probability.

How could I extend this to a general MDP? That is, an MDP where the sets $$\mathcal{A}$$, $$\mathcal{S}$$, and $$\mathcal{R}$$ haven't a finite number of elements? To be more precise, in my case $$\mathcal{A} \subset \mathbb{R}^n$$, $$\mathcal{S} \subset \mathbb{R}^m$$, and $$\mathcal{R} \subset \mathbb{R}$$. My thought is that the equation above is still true, however, the probability is zero for each tuple $$s',r,s,a$$.

Is it sufficient to say that for finite MDP we have

$$\sum_{s'\in\mathcal{S}}\sum_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$

while in non-finite MDP (supposing that the sets $$\mathcal{s}$$ and $$\mathcal{A}$$ are continuous) we have

$$\int_{s'\in\mathcal{S}}\int_{r\in\mathcal{R}}p(s',r|s,a)=1 ~~~ \forall s\in\mathcal{S},a\in\mathcal{A}$$

or is it more complex than this?