Assume the existence of a Markov Decision Process consisting of:
- State space $S$
- Action space $A$
- Transition model $T: S \times A \times S \to [0,1]$
- Reward function $R: S \times A \times S \to \mathbb{R}$
- Distribution of initial state $p_0: S \to [0,1]$
and a policy $\pi:S\to A$.
The V- and Q-functions take expectations of the sum of future rewards. Let's start off by $r_0:= R(x_0,\pi(x_0),x_1)$, where $\pi$ is the current policy while $x_0 \sim p_0$ and $x_1 \sim T(x_0,\pi(x_0),-)$ are random variables. With setting $\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$ I obtain
$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0$, where $\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$ is the pushforward of $\mu_i$ under random variable $\rho_i$. But the above quantity still depends on $x_0$ as both $\mu_o$ and $\rho_0$ depend on $x_0$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$ is that correct ?
Now, the $V-$ and $Q-$functions take the expectation over the sum $R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$, where the instant of termination $T$ itself is a random variable, and besides that the agent does not know its distribution it is not even included in the MDP model. How can I take the expectation over a sum where the number of summands is random. We cannot just calculate $\sum^{E[T]}(\dots)$, because $E[T]$ might not even be integer.
