8 votes
Accepted

Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$ where $R_{t+1}$ is the random variable that represents the reward ...
user avatar
4 votes
Accepted

Why is there an expectation sign in the Bellman equation?

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons- The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\...
user avatar
  • 78
4 votes

Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

David Ireland gives a fantastic answer, and I will provide an intuitive and gentle (but less rigorous) answer for those who are unfamiliar with the relevant statistical concepts. Next reward $R_{t+1}...
user avatar
  • 776
4 votes
Accepted

Why is the mean used to compute the expectation in the GAN loss?

It seems your question is concerned with how an empirical mean works. It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\...
user avatar
3 votes
Accepted

How is the state-value function expressed as a product of sums?

A quick review of resolving expectations: If you know that a discrete random variable $X$, drawn from set $\mathcal{X}$ has probability distribution $p(x) = \mathbf{Pr}\{X=x \}$, then $$\mathbb{E}[X] =...
user avatar
  • 23.2k
3 votes
Accepted

What is wrong with equation 7.3 in Sutton & Barto's book?

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are ...
user avatar
3 votes
Accepted

How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?

Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in $\mathbb{E}[X|Y]$, we always calculate the expected value ...
user avatar
3 votes
Accepted

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

First note that $\mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s)$ where $m(\cdot)$ is the mass function for the joint distribution of $S_{t+1},R_{t+1}$. If you are currently in state $S_t$ and we ...
user avatar
3 votes
Accepted

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
user avatar
  • 23.2k
3 votes
Accepted

Are these two definitions of the state-action value function equivalent?

The definition of the state-action value function is always the same. Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional ...
user avatar
  • 33k
3 votes

What does the argmax of the expectation of the log likelihood mean?

This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts): Some more explanation from same ...
user avatar
  • 927
2 votes
Accepted

What does the notation ${s'\sim T(s,a,\cdot)}$ mean?

The dot ($.$) at the end of $T(s,a,.)$ shows all possible states that we can go from state $S$ by doing action $a$. As you know there are some probabilities here for choosing those states, that the ...
user avatar
  • 1,663
2 votes
Accepted

How does the policy gradient's derivative work?

You cannot do this: $\mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )]$ That ...
user avatar
  • 23.2k
2 votes

Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks'

we start with $\frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)]$ using definition of expectation for continuous case: $\mathbb E[X] = \int xp(x) dx$ ...
user avatar
  • 2,216
2 votes

Why is there an expectation sign in the Bellman equation?

In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state),...
user avatar
  • 33k
2 votes

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Note that for a general policy $\pi$ we have that $q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]$, where in state $S_t$ we take action $a$ and thereafter following policy $\pi$. Note that ...
user avatar
1 vote

What are the iid random variables for a dataset in the GAN framework?

Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The ...
user avatar
1 vote
Accepted

What is the meaning of these equations in Noise2Noise paper?

The equation you are referring to is called Mean Squared Error (or $L_2$ loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs ...
user avatar
  • 600
1 vote
Accepted

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$? This is partly a notation issue, but $G_t$ is already the future sum of rewards as ...
user avatar
  • 23.2k
1 vote
Accepted

Why is the expectation calculated over finite number of points drawn from a probability distribution?

When we say that we have $N$ points that were "drawn from the probability distribution or probability density", this means that every point $x_n$ had the correct probability $p(x_n)$ of being sampled ...
user avatar
  • 9,316
1 vote

What is the difference between return and expected return?

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
user avatar
  • 111
1 vote

What is the difference between return and expected return?

Formally, the return (also known as the cumulative future discounted reward) can be defined as $$ G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}, $$ where $0 \leq \gamma \leq 1$ is the discount factor and ...
user avatar
  • 33k

Only top scored, non community-wiki answers of a minimum length are eligible