What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?

Question

I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $q_∗(s, a)$ for a MDP with continuous states and actions, written out using explicit integration (no expectation notation).

The discrete case is

$$ q_*(s,a) = \sum_{s'}\sum_r p(s',r|s,a)[r+\gamma \max_{a'}q*(s',a')] $$

My thoughts for the continuous case are:

\begin{align} q_*(s,a) &= \int_{s'}\int_{r}f_{s',r|s,a}(s',r|s,a)[r+\gamma\max_{a'}q_*(s',a')]drds' \end{align}

Is this how it would look like?

Answer 1

I think your equations are alright.

Anyway, this is just a question of mathematical notation.

In measure theory, a discrete random variable $X$ is said to have a counting measure over it's support $\mathcal X = \{x_k,k=1,...K\}$.

We can define its distribution function as $F_X(x)=\sum_k p(x_k) 1(x \ge x_k)$.

With the distribution function, we can write the expectation as the measure-theoretical integration $EX=\int _\mathcal X xF_X(dx)$.

Notice the part $F_X(dx)$, which is not the usual $dF_X(x)$ notation for Riemann integration.

This is why many graduate-level texts prefer to measure theory to define integration and expectations - so that we can use one unified notation for discrete random variables and continuous random variables, and any random variable you can think of living in some abstract measure space $(\Omega, \mathcal F, \mathbb P)$.