8
votes
Accepted
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
Let's first write the state-value function as
$$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$
where $R_{t+1}$ is the random variable that represents the reward ...
- 4,197
4
votes
Accepted
Why is there an expectation sign in the Bellman equation?
There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-
The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\...
- 78
4
votes
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts.
Next reward $R_{t+1}...
- 1,027
4
votes
Accepted
Why is the mean used to compute the expectation in the GAN loss?
It seems your question is concerned with how an empirical mean works.
It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\...
- 4,197
3
votes
Accepted
How is the state-value function expressed as a product of sums?
A quick review of resolving expectations: If you know that a discrete random variable $X$, drawn from set $\mathcal{X}$ has probability distribution $p(x) = \mathbf{Pr}\{X=x \}$, then
$$\mathbb{E}[X] =...
- 24.7k
3
votes
Accepted
What is wrong with equation 7.3 in Sutton & Barto's book?
In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are ...
- 4,197
3
votes
Accepted
How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?
Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value ...
- 4,197
3
votes
Accepted
If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$.
If you are currently in state $S_t$ and we ...
- 4,197
3
votes
Accepted
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$?
The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
- 24.7k
3
votes
Accepted
Are these two definitions of the state-action value function equivalent?
The definition of the state-action value function is always the same.
Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
- 35k
3
votes
What does the argmax of the expectation of the log likelihood mean?
This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts):
Some more explanation from same ...
- 927
2
votes
Accepted
What does the notation ${s'\sim T(s,a,\cdot)}$ mean?
The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the ...
- 1,683
2
votes
Accepted
How does the policy gradient's derivative work?
You cannot do this:
$\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$
That ...
- 24.7k
2
votes
Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks'
we start with
$\frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)]$
using definition of expectation for continuous case:
$\mathbb E[X] = \int xp(x) dx$
...
- 2,236
2
votes
Why is there an expectation sign in the Bellman equation?
In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state),...
- 35k
2
votes
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Note that for a general policy $\pi$ we have that $q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]$, where in state $S_t$ we take action $a$ and thereafter following policy $\pi$. Note that ...
- 4,197
2
votes
Accepted
How to correctly evaluate the state value of this simple markov decision process?
The issue is with your calculation of $R_{a-b-a}$: you're missing the +1 reward on the path from B to A.
2
votes
Accepted
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
Question: Can I write it without the subscript? So $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$
Yes, your reasoning is sound, there is no need to condition the expectation on the policy, ...
- 24.7k
1
vote
What are the iid random variables for a dataset in the GAN framework?
Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The ...
- 772
1
vote
Accepted
What is the meaning of these equations in Noise2Noise paper?
The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input.
In your case, the inputs ...
- 600
1
vote
Accepted
Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$?
This is partly a notation issue, but $G_t$ is already the future sum of rewards as ...
- 24.7k
1
vote
Accepted
Why is the expectation calculated over finite number of points drawn from a probability distribution?
When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled ...
- 9,519
1
vote
What is the difference between return and expected return?
You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
- 111
1
vote
What is the difference between return and expected return?
Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and ...
- 35k
Only top scored, non community-wiki answers of a minimum length are eligible