# Tag Info

4

It seems your question is concerned with how an empirical mean works. It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\lim_{n \rightarrow \infty} \frac{1}{n}\sum_{i=1}^n f(x^{(i)}) = \mathbb{E}[f(X)]$. This is a standard result in statistics known as the law of large numbers.

3

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the $n$th step return. They would only be equal as $n \rightarrow \infty$.

3

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible future returns whilst $v_*(S_{t+1})$ is a probability distribution derived over all possible values of $S_{t+1}$. These will be different distributions much of the ...

3

These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation. The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by $\theta$ (i.e. the policy could be represented e.g. by a neural network), ...

3

Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value over distribution $X$. No, for $\mathbb{E}[X|Y]$ we take expectation of $X$ with respect to the conditional distribution $X|Y$, i.e. $$\mathbb{E}[X|Y] = \... 2 Note that for a general policy \pi we have that q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a], where in state S_t we take action a and thereafter following policy \pi. Note that the expectation is taken with respect to the reward transition distribution \mathbb{P}(R_{t+1} = r, S_{t+1} = s' | A_t = a, S_t = s) which I will denote as p(s'... 1 The equation you are referring to is called Mean Squared Error (or L_2 loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs are measurements of temperature y, either at a certain point in time or point in space or both or none, this is not clear from the image. Now, the goal would ... 1 As mentioned in the comments your assumption about independence is wrong. Here's why. To prove independence we need to show the following holds:$$P(X=x, Y=y) = P(X=x)P(Y=y)$$in the case of RL this becomes:$$P(X=a, X=a') = P(X=a)P(Y=a')$$The left hand side has the value:$$P(X=a, Y=a') = b(A_t = a| S_t = s) p(s'|a,s) b(A_{t+1} = a'|, S_{t+1} = s') while ...

1

shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$? This is partly a notation issue, but $G_t$ is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g_t$* until after $t+n$. However, you ...

Only top voted, non community-wiki answers of a minimum length are eligible