Questions tagged [expectation]
For questions related to the mathematical concept of "expectation" or "expected value".
25
questions
1
vote
1
answer
34
views
How to correctly evaluate the state value of this simple markov decision process?
For some contexts, I'm working on a c# library for reinforcement learning. I implemented two methods to evaluate a state value function, namely the TD(0) method and the Monte Carlo first visit method.
...
- 113
4
votes
1
answer
53
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
- 81
0
votes
0
answers
13
views
How is the union bound being used?
I am trying to understand the assumption proof of Theorem 2(Page -$7$) in the paper "A Universal Law of Robustness via isoperimetry" by Bubeck and Sellke.
Inequality 1
\begin{align}
\mathbb{...
- 111
1
vote
1
answer
71
views
What are the iid random variables for a dataset in the GAN framework?
I am trying to understand why mean is used for expectation in training Generative Adversarial Networks.
The answer tells that it is due to the law of large numbers which is based on the assumption ...
- 3,251
4
votes
1
answer
221
views
How is the state-value function expressed as a product of sums?
The state-value function for a given policy $\pi$ is given by
$$\begin{align}
V^{\pi}(s) &=E_{\pi}\left\{r_{t+1}+\gamma r_{t+2}+\gamma^{2} r_{t+3}+\cdots \mid s_{t}=s\right\} \\
&=E_{\pi}\...
- 3,251
0
votes
1
answer
197
views
What is the meaning of these equations in Noise2Noise paper?
I am trying to understand what is meant by following equations in the Noise2Noise paper by Nvidia.
What is meant by the equation in this image? What is $\mathbb{E}_y\{y\}$? And how should I try to ...
- 31
3
votes
1
answer
131
views
What is wrong with equation 7.3 in Sutton & Barto's book?
Equation 7.3 of Sutton Barto book:
$$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$
$$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}...
- 147
5
votes
1
answer
103
views
Why is the mean used to compute the expectation in the GAN loss?
From Goodfellow et al. (2014), we have the adversarial loss:
$$ \min_G \, \max_D V (D, G) = \mathbb{E}_{x∼p_{data}(x)} \, [\log \, D(x)] + \, \mathbb{E}_{z∼p_z(z)} \, [\log \, (1 − D(G(z)))] \, \text{...
2
votes
2
answers
571
views
How is per-decision importance sampling derived in Sutton & Barto's book?
In per-decison importance sampling given in Sutton & Barto's book:
Eq 5.12 $\rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(...
- 147
0
votes
0
answers
97
views
Problem in understanding equation given for convergence of TD(n) algorithm
Given equation 7.3 of Sutton and Barto's book for convergence of TD(n):
$\max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi(s)| \leqslant \gamma^n \max_s|V_{t+n-1}(s) - v_\pi(s)|$
$\textbf{PROBLEM ...
- 147
3
votes
1
answer
94
views
How does $\mathbb{E}$ suddenly change to $\mathbb{E}_{\pi'}$ in this equation?
In Sutton-Barto's book on page 63 (81 of the pdf):
$$\mathbb{E}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_t=s,A_t=\pi'(s)] = \mathbb{E}_{\pi'}[R_{t+1} + \gamma v_\pi(S_{t+1}) \mid S_{t} = s]$$
How does $...
- 147
7
votes
2
answers
364
views
Why does the state-action value function, defined as an expected value of the reward and state value function, not need to follow a policy?
I often see that the state-action value function is expressed as:
$$q_{\pi}(s,a)=\color{red}{\mathbb{E}_{\pi}}[R_{t+1}+\gamma G_{t+1} | S_t=s, A_t = a] = \color{blue}{\mathbb{E}}[R_{t+1}+\gamma v_{\pi}...
- 313
5
votes
1
answer
631
views
If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
I'm trying to solve exercise 3.11 from the book Sutton and Barto's book (2nd edition)
Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, ...
- 362
4
votes
2
answers
127
views
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
In equation 3.17 of Sutton and Barto's book:
$$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$
$G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
- 147
2
votes
1
answer
178
views
Are these two definitions of the state-action value function equivalent?
I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function ...
- 4,197
3
votes
1
answer
75
views
What is meant by the expected BLEU cost when training with BLEU and SIMILE?
Recently I was reading a paper based on a new evaluation metric SIMILE. In a section, validation loss comparison had been made for SIMILE and BLEU. The plot showed the expected BLEU cost when training ...
- 31
3
votes
1
answer
48
views
Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
- 341
2
votes
2
answers
104
views
Why is there an expectation sign in the Bellman equation?
In chapter 3.5 of Sutton's book, the value function is defined as:
Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the ...
- 23
2
votes
1
answer
45
views
What does the notation ${s'\sim T(s,a,\cdot)}$ mean?
I have been seeing notations on Expectations with their respective subscripts such as $E_{s_0 \sim D}[V^\pi (s_0)] = \Sigma_{t=0}^\infty[\gamma^t\phi(s_t)]$. This equation is taken from https://ai....
- 1,241
1
vote
1
answer
349
views
How does the policy gradient's derivative work?
I am trying to understand the policy gradient method using a PyTorch implementation and this tutorial.
My first question is about the end result of this gradient derivation,
\begin{aligned}
\nabla \...
- 956
2
votes
1
answer
54
views
Why is the expectation calculated over finite number of points drawn from a probability distribution?
This is from the book Pattern Recognition by Bishop. Why is expectation here a simple average? Why is $f(x)$ not being multiplied by $p(x)$?
- 113
4
votes
2
answers
322
views
What is the difference between return and expected return?
At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.
If an agent is following a policy (which in itself is a probability ...
- 41
1
vote
1
answer
98
views
Problem with Proposition 1 of Google Deepmind's 'Weight uncertainty in Neural Networks'
I'm going through the paper Weight Uncertainty in Neural Networks by Google Deepmind. In the final line of the proof of proposition 1, the integral and the derivative are swapped. Then the derivative ...
- 21
1
vote
0
answers
49
views
Where does the expectation term in the derivative of the soft-max policy come from?
At slide 17 of the David Silver's series, the soft-max policy is defined as follows
$$
\pi_\theta(s, a) \propto e^{\phi(s, a)^T \theta}
$$
that is, the probability of an action $a$ (in state $s$) is ...
- 253
5
votes
2
answers
997
views
What does the argmax of the expectation of the log likelihood mean?
What does the following equation mean? What does each part of the formula represent or mean?
$$\theta^* = \underset {\theta}{\arg \max} \Bbb E_{x \sim p_{data}} \log {p_{model}(x|\theta) }$$
- 181