# What would be the Bellman optimality equation for $q_∗(s, a)$ for an MDP with continuous states and actions?

I'm currently studying Reinforcement Learning and I'd like to know what would be the Bellman optimality equation for action values $$q_∗(s, a)$$ for a MDP with continuous states and actions, written out using explicit integration (no expectation notation).

The discrete case is

$$q_*(s,a) = \sum_{s'}\sum_r p(s',r|s,a)[r+\gamma \max_{a'}q*(s',a')]$$

My thoughts for the continuous case are:

\begin{align} q_*(s,a) &= \int_{s'}\int_{r}f_{s',r|s,a}(s',r|s,a)[r+\gamma\max_{a'}q_*(s',a')]drds' \end{align}

Is this how it would look like?

I think your equations are alright.

Anyway, this is just a question of mathematical notation.

In measure theory, a discrete random variable $$X$$ is said to have a counting measure over it's support $$\mathcal X = \{x_k,k=1,...K\}$$.

We can define its distribution function as $$F_X(x)=\sum_k p(x_k) 1(x \ge x_k)$$.

With the distribution function, we can write the expectation as the measure-theoretical integration $$EX=\int _\mathcal X xF_X(dx)$$.

Notice the part $$F_X(dx)$$, which is not the usual $$dF_X(x)$$ notation for Riemann integration.

This is why many graduate-level texts prefer to measure theory to define integration and expectations - so that we can use one unified notation for discrete random variables and continuous random variables, and any random variable you can think of living in some abstract measure space $$(\Omega, \mathcal F, \mathbb P)$$.