# Questions tagged [expectation]

For questions related to the mathematical concept of "expectation" or "expected value".

21 questions
Filter by
Sorted by
Tagged with
71 views

111 views

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)|$$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}... 1answer 90 views ### How does \mathbb{E} suddenly change to \mathbb{E}_{\pi'} in this equation? In Sutton-Barto's book on page 63 (81 of the pdf):$$\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s,A_t=\pi'(s)] = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_{t} = s]How does ... 1answer 42 views ### Shouldn't expected return be calculated for some faraway time in the future t+n instead of current time t? I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ... 1answer 56 views ### What is meant by the expected BLEU cost when training with BLEU and SIMILE? Recently I was reading a paper based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training ... 2answers 67 views ### Why is there an expectation sign in the Bellman equation? In chapter 3.5 of Sutton's book, the value function is defined as: Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ... 1answer 111 views ### Are these two definitions of the state-action value function equivalent? I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ... 1answer 40 views ### What does the notation {s'\sim T(s,a,\cdot)} mean? I have been seeing notations on Expectations with their respective subscripts such as E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]. This equation is taken from https://ai.... 2answers 397 views ### How is per-decision importance sampling derived in Sutton & Barto's book? In per-decison importance sampling given in Sutton & Barto's book: Eq 5.12 \rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(... 1answer 47 views ### Why is the expectation calculated over finite number of points drawn from a probability distribution? This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is f(x) not being multiplied by p(x)? 1answer 206 views ### How does the policy gradient's derivative work? I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial. My first question is about the end result of this gradient derivation, \begin{aligned} \nabla \... 1answer 80 views ### Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks' I'm going through the paper Weight Uncertainty in Neural Networks by Google Deepmind. In the final line of the proof of proposition 1, the integral and the derivative are swapped. Then the derivative ... 1answer 38 views ### What are the iid random variables for a dataset in the GAN framework? I am trying to understand why mean is used for expectation in training Generative Adversarial Networks. The answer tells that it is due to the law of large numbers which is based on the assumption ... 0answers 43 views ### Where does the expectation term in the derivative of the soft-max policy come from? At slide 17 of the David Silver's series, the soft-max policy is defined as follows \pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}  that is, the probability of an action $a$ (in state $s$) is ...
I am trying to understand what is meant by following equations in the Noise2Noise paper by Nvidia. What is meant by the equation in this image? What is $\mathbb{E}_y\{y\}$? And how should I try to ...
Given equation 7.3 of Sutton and Barto's book for convergence of TD(n): $\max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi(s)| \leqslant \gamma^n \max_s|V_{t+n-1}(s) - v_\pi(s)|$ \$\textbf{PROBLEM ...