Assume the existence of a Markov Decision Process consisting of:

  • State space $S$
  • Action space $A$
  • Transition model $T: S \times A \times S \to [0,1]$
  • Reward function $R: S \times A \times S \to \mathbb{R}$
  • Distribution of initial state $p_0: S \to [0,1]$

and a policy $\pi:S\to A$.

The V- and Q-functions take expectations of the sum of future rewards. Let's start off by $r_0:= R(x_0,\pi(x_0),x_1)$, where $\pi$ is the current policy while $x_0 \sim p_0$ and $x_1 \sim T(x_0,\pi(x_0),-)$ are random variables. With setting $\mu_i:= T(x_i,\pi(x_i),-),\rho_i:=R(x_i,\pi(x_i),-)$ I obtain

$E[r_0]= \int_{\mathbb{R}} r d\mu_0^{\rho_0} = \int_S R(x_0,\pi(x_0),-)d\mu_0$, where $\mu_i^{\rho_i}:= \mu_i\circ \rho_i^{-1}$ is the pushforward of $\mu_i$ under random variable $\rho_i$. But the above quantity still depends on $x_0$ as both $\mu_o$ and $\rho_0$ depend on $x_0$. Intuitively, I would guess that one has to calculate the integral over every occuring random variable to obtain the overall expectation, i.e. $E[r_0]= \int_S\int_S R(x_0,\pi(x_0),x_1)d\mu_0(x_1)dp_0(x_0)$ is that correct ?

Now, the $V-$ and $Q-$functions take the expectation over the sum $R_{\tau} = \sum^T_{t=\tau}\gamma^{t-\tau}r_t$, where the instant of termination $T$ itself is a random variable, and besides that the agent does not know its distribution it is not even included in the MDP model. How can I take the expectation over a sum where the number of summands is random. We cannot just calculate $\sum^{E[T]}(\dots)$, because $E[T]$ might not even be integer.

  • $\begingroup$ Hi. Can you please simplify your question? Ask one question per post, if you want to attract the attention of more people. It seems to me you're asking multiple questions here. $\endgroup$ – nbro Feb 18 '20 at 22:00
  • $\begingroup$ For the question related to $V$ and $Q$ I feel like you are overthinking it (either that, or I don't understand your question). The sum of $R_\tau$ is a random variable so it is fine for it to have an expectation. You can circumvent the problem of $E[T]$ (the length of the trajectory) by just assuming, that the trajectory is infinite and that after $T$ the reward is zero. $\endgroup$ – Hai Nguyen Feb 19 '20 at 9:09
  • $\begingroup$ @nbro Yes, I will split it into serveral questions. $\endgroup$ – Leroy Od Feb 26 '20 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.