0
$\begingroup$

Assume the existence of a Markov Decision Process consisting of:

  • State space $S$
  • Action space $A$
  • Transition model $T: S \times A \times S \to [0,1]$
  • Reward function $R: S \times A \times S \to \mathbb{R}$
  • Distribution of initial state $p_0: S \to [0,1]$

and a policy $\pi: S \to A$.

The $V$ and $Q$-functions take expectations of the sum of future rewards.

Let's start off by $r_0:= R(x_0,\pi(x_0),x_1)$, where $\pi$ is the current policy while $x_0 \sim p_0$ and $x_1 \sim T(x_0,\pi(x_0),-)$ are random variables. With setting $\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$, I obtain

$$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0,$$

where $\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$ is the pushforward of $\mu_i$ under random variable $\rho_i$. But the above quantity still depends on $x_0$ as both $\mu_o$ and $\rho_0$ depend on $x_0$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $$E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$$ is that correct ?

Now, the $V$ and $Q$-functions take the expectation over the sum $R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$, where the instant of termination $T$ itself is a random variable, and, besides that, the agent does not know its distribution, as it is not even included in the MDP model.

How can I take the expectation over a sum where the number of summands is random?

We cannot just calculate $\sum^{E[T]}(\dots)$, because $E[T]$ might not even be an integer.

$\endgroup$
3
  • $\begingroup$ Hi. Can you please simplify your question? Ask one question per post, if you want to attract the attention of more people. It seems to me you're asking multiple questions here. $\endgroup$
    – nbro
    Feb 18 '20 at 22:00
  • 1
    $\begingroup$ For the question related to $V$ and $Q$ I feel like you are overthinking it (either that, or I don't understand your question). The sum of $R_\tau$ is a random variable so it is fine for it to have an expectation. You can circumvent the problem of $E[T]$ (the length of the trajectory) by just assuming, that the trajectory is infinite and that after $T$ the reward is zero. $\endgroup$
    – Hai Nguyen
    Feb 19 '20 at 9:09
  • $\begingroup$ @nbro Yes, I will split it into serveral questions. $\endgroup$
    – Leroy Od
    Feb 26 '20 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.