# Tag Info

## Hot answers tagged expectation

8 votes
Accepted

### Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?

Let's first write the state-value function as $$q_{\pi}(s,a) = \mathbb{E}_{p, \pi}[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]\;,$$ where $R_{t+1}$ is the random variable that represents the reward ...
• 4,026
4 votes
Accepted

• 776
4 votes
Accepted

It seems your question is concerned with how an empirical mean works. It is indeed true that, if all $x^{(i)}$ are independent identically distributed realisations of a random variable $X$, then $\... • 4,026 3 votes Accepted ### How is the state-value function expressed as a product of sums? A quick review of resolving expectations: If you know that a discrete random variable$X$, drawn from set$\mathcal{X}$has probability distribution$p(x) = \mathbf{Pr}\{X=x \}$, then $$\mathbb{E}[X] =... • 23.2k 3 votes Accepted ### What is wrong with equation 7.3 in Sutton & Barto's book? In general, \mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s). v_\pi(s) is defined as \mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s], so you should be able to see why the two are ... • 4,026 3 votes Accepted ### How does \mathbb{E} suddenly change to \mathbb{E}_{\pi'} in this equation? Also, in general, in the conditional expectation, which distribution do we compute the expectation with respect to? From what I have seen, in \mathbb{E}[X|Y], we always calculate the expected value ... • 4,026 3 votes Accepted ### If the current state is S_t and the actions are chosen according to \pi, what is the expectation of R_{t+1} in terms of \pi and p? First note that \mathbb{E}[R_{t+1} |S_t=s] = \sum_{s',r}rm(s',r|s) where m(\cdot) is the mass function for the joint distribution of S_{t+1},R_{t+1}. If you are currently in state S_t and we ... • 4,026 3 votes Accepted ### Why is G_{t+1} is replaced with v_*(S_{t+1}) in the Bellman optimality equation? Can someone provide the reasoning behind why G_{t+1} is equal to v_*(S_{t+1})? The two things are not usually exactly equal, because G_{t+1} is a probability distribution over all possible ... • 23.2k 3 votes Accepted ### Are these two definitions of the state-action value function equivalent? The definition of the state-action value function is always the same. Your definition is correct, as q_{\pi}(s,a) is conditioned on a, so you don't need to write q_{\pi}(s,a) as an conditional ... • 33k 3 votes ### What does the argmax of the expectation of the log likelihood mean? This equation and more information of it can be found in Expectation Maximization Wikipedia site and the explanation there was as follows (formula there in two parts): Some more explanation from same ... • 927 2 votes Accepted ### What does the notation {s'\sim T(s,a,\cdot)} mean? The dot (.) at the end of T(s,a,.) shows all possible states that we can go from state S by doing action a. As you know there are some probabilities here for choosing those states, that the ... • 1,663 2 votes Accepted ### How does the policy gradient's derivative work? You cannot do this: \mathop{\mathbb{E}_\pi }[r(\tau )\bigtriangledown log \pi (\tau )] \\= \mathop{\mathbb{E}_\pi }[r(\tau )] \,\, \mathop{\mathbb{E}_\pi }[\bigtriangledown log \pi (\tau )] That ... • 23.2k 2 votes ### Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks' we start with \frac{\partial}{\partial \theta} \mathbb E_{q(\mathbf w\mid\theta)}[f(\mathbf w, \theta)] using definition of expectation for continuous case: \mathbb E[X] = \int xp(x) dx ... • 2,216 2 votes ### Why is there an expectation sign in the Bellman equation? In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state),... • 33k 2 votes ### Why is G_{t+1} is replaced with v_*(S_{t+1}) in the Bellman optimality equation? Note that for a general policy \pi we have that q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a], where in state S_t we take action a and thereafter following policy \pi. Note that ... • 4,026 1 vote ### What are the iid random variables for a dataset in the GAN framework? Independent and identically distributed random variables share the same probability distribution and each item doesn’t influence or provide insight about the value of the next item you measure. The ... • 732 1 vote Accepted ### What is the meaning of these equations in Noise2Noise paper? The equation you are referring to is called Mean Squared Error (or L_2 loss) and it is used for regression tasks, where the goal is to predict a real value given some input. In your case, the inputs ... • 600 1 vote Accepted ### Shouldn't expected return be calculated for some faraway time in the future t+n instead of current time t? shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time t? This is partly a notation issue, but G_t is already the future sum of rewards as ... • 23.2k 1 vote Accepted ### Why is the expectation calculated over finite number of points drawn from a probability distribution? When we say that we have N points that were "drawn from the probability distribution or probability density", this means that every point x_n had the correct probability p(x_n) of being sampled ... • 9,316 1 vote ### What is the difference between return and expected return? You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations. • 111 1 vote ### What is the difference between return and expected return? Formally, the return (also known as the cumulative future discounted reward) can be defined as$$ G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},$$where$0 \leq \gamma \leq 1\$ is the discount factor and ...
• 33k

Only top scored, non community-wiki answers of a minimum length are eligible