# Understanding V- and Q-functions

Assume the existence of a Markov Decision Process consisting of:

• State space $$S$$
• Action space $$A$$
• Transition model $$T: S \times A \times S \to [0,1]$$
• Reward function $$R: S \times A \times S \to \mathbb{R}$$
• Distribution of initial state $$p_0: S \to [0,1]$$

and a policy $$\pi:S\to A$$.

The V- and Q-functions take expectations of the sum of future rewards. Let's start off by $$r_0:= R(x_0,\pi(x_0),x_1)$$, where $$\pi$$ is the current policy while $$x_0 \sim p_0$$ and $$x_1 \sim T(x_0,\pi(x_0),-)$$ are random variables. With setting $$\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$$ I obtain

$$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0$$, where $$\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$$ is the pushforward of $$\mu_i$$ under random variable $$\rho_i$$. But the above quantity still depends on $$x_0$$ as both $$\mu_o$$ and $$\rho_0$$ depend on $$x_0$$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $$E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$$ is that correct ?

Now, the $$V-$$ and $$Q-$$functions take the expectation over the sum $$R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$$, where the instant of termination $$T$$ itself is a random variable, and besides that the agent does not know its distribution it is not even included in the MDP model. How can I take the expectation over a sum where the number of summands is random. We cannot just calculate $$\sum^{E[T]}(\dots)$$, because $$E[T]$$ might not even be integer.

• Hi. Can you please simplify your question? Ask one question per post, if you want to attract the attention of more people. It seems to me you're asking multiple questions here. – nbro Feb 18 '20 at 22:00
• For the question related to $V$ and $Q$ I feel like you are overthinking it (either that, or I don't understand your question). The sum of $R_\tau$ is a random variable so it is fine for it to have an expectation. You can circumvent the problem of $E[T]$ (the length of the trajectory) by just assuming, that the trajectory is infinite and that after $T$ the reward is zero. – Hai Nguyen Feb 19 '20 at 9:09
• @nbro Yes, I will split it into serveral questions. – Leroy Od Feb 26 '20 at 22:16