# Tag Info

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons- The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $... 4 It seems your question is concerned with how an empirical mean works. It is indeed true that, if all$x^{(i)}$are independent identically distributed realisations of a random variable$X$, then$\lim_{n \rightarrow \infty} \frac{1}{n}\sum_{i=1}^n f(x^{(i)}) = \mathbb{E}[f(X)]$. This is a standard result in statistics known as the law of large numbers. 3 In general,$\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$.$v_\pi(s)$is defined as$\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the$n$th step return. They would only be equal as$n \rightarrow \infty$. 3 Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in$\mathbb{E}[X|Y]$, we always calculate the expected value over distribution$X$. No, for$\mathbb{E}[X|Y]$we take expectation of$X$with respect to the conditional distribution$X|Y$, i.e. $$\mathbb{E}[X|Y] = \... 3 Can someone provide the reasoning behind why G_{t+1} is equal to v_*(S_{t+1})? The two things are not usually exactly equal, because G_{t+1} is a probability distribution over all possible future returns whilst v_*(S_{t+1}) is a probability distribution derived over all possible values of S_{t+1}. These will be different distributions much of the ... 3 These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation. The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by \theta (i.e. the policy could be represented e.g. by a neural network), ... 2 The dot (.) at the end of T(s,a,.) shows all possible states that we can go from state S by doing action a. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, T(s,a,.) is a probability distribution. 2 You cannot do this: \mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )] That is because r(\tau ) and \bigtriangledown log \pi (\tau ) are correlated by their dependence on \tau. In a simpler concrete example, if your expectation ... 2 we start with \frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)] using definition of expectation for continuous case: \mathbb E[X] = \int xp(x) dx for the first equation we get: \frac{\partial}{\partial \theta} \int f(\mathbf w, \theta)q(\mathbf w \mid \theta) d\mathbf w we swap q(\mathbf w \mid \... 2 This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts): Some more explanation from same page: In statistics, an expectation–maximization (EM) algorithm is an iterative method to find maximum likelihood or maximum a posteriori (MAP) estimates of ... 2 Note that for a general policy \pi we have that q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a], where in state S_t we take action a and thereafter following policy \pi. Note that the expectation is taken with respect to the reward transition distribution \mathbb{P}(R_{t+1} = r, S_{t+1} = s' | A_t = a, S_t = s) which I will denote as p(s'... 2 In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant ... 2 A quick review of resolving expectations: If you know that a discrete random variable X, drawn from set \mathcal{X} has probability distribution p(x) = \mathbf{Pr}\{X=x \}, then$$\mathbb{E}[X] = \sum_{x \in \mathcal{X}} xp(x)$$This equation is the core of what is going on when resolving the expectation in your quoted equation. Resolving the ... 1 The equation you are referring to is called Mean Squared Error (or L_2 loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs are measurements of temperature y, either at a certain point in time or point in space or both or none, this is not clear from the image. Now, the goal would ... 1 As mentioned in the comments your assumption about independence is wrong. Here's why. To prove independence we need to show the following holds:$$P(X=x, Y=y) = P(X=x)P(Y=y)$$in the case of RL this becomes:$$P(X=a, X=a') = P(X=a)P(Y=a')$$The left hand side has the value:$$P(X=a, Y=a') = b(A_t = a| S_t = s) p(s'|a,s) b(A_{t+1} = a'|, S_{t+1} = s')$$while ... 1 shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time t? This is partly a notation issue, but G_t is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return g_t* until after t+n. However, you ... 1 When we say that we have N points that were "drawn from the probability distribution or probability density", this means that every point x_n had the correct probability p(x_n) of being sampled from the distribution when we were sampling our n^{th} point. For example, suppose that we wish to compute/estimate the expected value of a distribution ... 1 You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations. 1 Formally, the return (also known as the cumulative future discounted reward) can be defined as$$ G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},$$where$0 \leq \gamma \leq 1$is the discount factor and$R_{i}$is the reward at time step$i$. Here$G_t$and$R_i\$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using ...