4
There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-
The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $...
4
It seems your question is concerned with how an empirical mean works.
It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\lim_{n \rightarrow \infty} \frac{1}{n}\sum_{i=1}^n f(x^{(i)}) = \mathbb{E}[f(X)]$. This is a standard result in statistics known as the law of large numbers.
3
In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are not equal when the LHS is an expectation of the $n$th step return. They would only be equal as $n \rightarrow \infty$.
3
Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value over distribution $X$.
No, for $\mathbb{E}[X|Y]$ we take expectation of $X$ with respect to the conditional distribution $X|Y$, i.e.
$$\mathbb{E}[X|Y] = \...
3
Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$?
The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible future returns whilst $v_*(S_{t+1})$ is a probability distribution derived over all possible values of $S_{t+1}$. These will be different distributions much of the ...
3
These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation.
The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by $\theta$ (i.e. the policy could be represented e.g. by a neural network), ...
3
we start with
$\frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)]$
using definition of expectation for continuous case:
$\mathbb E[X] = \int xp(x) dx$
for the first equation we get:
$\frac{\partial}{\partial \theta} \int f(\mathbf w, \theta)q(\mathbf w \mid \theta) d\mathbf w$
we swap $q(\mathbf w \mid \...
2
The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the sum of these probabilities is equal to 1. Hence, $T(s,a,.)$ is a probability distribution.
2
You cannot do this:
$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$
That is because $r(\tau )$ and $\bigtriangledown log \pi (\tau )$ are correlated by their dependence on $\tau$. In a simpler concrete example, if your expectation ...
2
This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts):
Some more explanation from same page:
In statistics, an expectation–maximization (EM) algorithm is an iterative method to find maximum likelihood or maximum a posteriori (MAP) estimates of ...
2
Note that for a general policy $\pi$ we have that $q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]$, where in state $S_t$ we take action $a$ and thereafter following policy $\pi$. Note that the expectation is taken with respect to the reward transition distribution $\mathbb{P}(R_{t+1} = r, S_{t+1} = s' | A_t = a, S_t = s)$ which I will denote as $p(s'...
2
In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant ...
1
The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input.
In your case, the inputs are measurements of temperature $y$, either at a certain point in time or point in space or both or none, this is not clear from the image. Now, the goal would ...
1
As mentioned in the comments your assumption about independence is wrong. Here's why. To prove independence we need to show the following holds:
$$P(X=x, Y=y) = P(X=x)P(Y=y)$$
in the case of RL this becomes:
$$P(X=a, X=a') = P(X=a)P(Y=a')$$
The left hand side has the value:
$$P(X=a, Y=a') = b(A_t = a| S_t = s) p(s'|a,s) b(A_{t+1} = a'|, S_{t+1} = s')$$
while ...
1
shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$?
This is partly a notation issue, but $G_t$ is already the future sum of rewards as seen by the first (and correct) equation in your question. You don't actually know the value of any individual return $g_t$* until after $t+n$. However, you ...
1
When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled from the distribution when we were sampling our $n^{th}$ point.
For example, suppose that we wish to compute/estimate the expected value of a distribution ...
1
You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
1
Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and $R_{i}$ is the reward at time step $i$. Here $G_t$ and $R_i$ are considered random variables (and r.v.s are usually denoted with capital letters, so I am using ...
Only top voted, non community-wiki answers of a minimum length are eligible
