Assume the existence of a Markov Decision Process consisting of:

  • State space $S$
  • Action space $A$
  • Transition model $T: S \times A \times S \to [0,1]$
  • Reward function $R: S \times A \times S \to \mathbb{R}$
  • Distribution of initial state $p_0: S \to [0,1]$

and a policy $\pi: S \to A$.

The $V$ and $Q$-functions take expectations of the sum of future rewards.

Let's start off by $r_0:= R(x_0,\pi(x_0),x_1)$, where $\pi$ is the current policy while $x_0 \sim p_0$ and $x_1 \sim T(x_0,\pi(x_0),-)$ are random variables. With setting $\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$, I obtain

$$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0,$$

where $\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$ is the pushforward of $\mu_i$ under random variable $\rho_i$. But the above quantity still depends on $x_0$ as both $\mu_o$ and $\rho_0$ depend on $x_0$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $$E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$$ is that correct ?

Now, the $V$ and $Q$-functions take the expectation over the sum $R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$, where the instant of termination $T$ itself is a random variable, and, besides that, the agent does not know its distribution, as it is not even included in the MDP model.

How can I take the expectation over a sum where the number of summands is random?

We cannot just calculate $\sum^{E[T]}(\dots)$, because $E[T]$ might not even be an integer.

  • $\begingroup$ Hi. Can you please simplify your question? Ask one question per post, if you want to attract the attention of more people. It seems to me you're asking multiple questions here. $\endgroup$
    – nbro
    Commented Feb 18, 2020 at 22:00
  • 1
    $\begingroup$ For the question related to $V$ and $Q$ I feel like you are overthinking it (either that, or I don't understand your question). The sum of $R_\tau$ is a random variable so it is fine for it to have an expectation. You can circumvent the problem of $E[T]$ (the length of the trajectory) by just assuming, that the trajectory is infinite and that after $T$ the reward is zero. $\endgroup$
    – Hai Nguyen
    Commented Feb 19, 2020 at 9:09
  • $\begingroup$ @nbro Yes, I will split it into serveral questions. $\endgroup$
    – Leroy Od
    Commented Feb 26, 2020 at 22:16


You must log in to answer this question.