2
$\begingroup$

In chapter 3.5 of Sutton's book, the value function is defined as:

enter image description here

Can someone give me some clarification about why there is the expectation sign behind the entire equation? Considering that the agent is following a fixed policy $\pi$, why there should be an expectation when the trajectory of the future possible states is fixed (or maybe I am getting it wrong and it's not). In total, if the expectation here has the meaning of averaging over a series of trajectories, what are those trajectories and what are the weights of them when we want to compute the expected value over them according to this Wikipedia definition of the expected values?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

There needs to be an $E_{\pi}$ over the infinite discounted return term because of two reasons-

  1. The policy could be stochastic in nature. That is, for any given state $s_t$ at time $t$, the policy $\pi(s_t)$ does not provide a deterministic action $a$, but rather, it provides us with a distribution over the possible next states, that is the action at time $t$, $a_t$ is distributed as- $$a_t \sim \pi(s_t)$$
  2. Even if the policy $\pi$ being followed by an agent is deterministic, there still needs to be an expectation over the behavior of the underlying stochastic MDP environment. That is, any action $a_t$, in general, only provides us with a distribution over the possible next states of the agent. That is, $$P(s_{t + 1} = s') = P_{\pi}(s' | s_t) = \sum_{a \in A} T(s,a,s') \times P_{\pi}(a_t = a)$$ Here $T(s, a, s')$ is the transition function for the MDP and the above equation captures the stochasticity arising from both 1 and 2.

As you see the expectation does not have to do with averaging over a collection of trajectories. However, that idea is often used in Monte-Carlo estimation of value functions.

EDIT: As pointed out in the comments, it is not correct to say that the expectation is not over a collection of trajectories.

Improve this answer
$\endgroup$
6
  • 4
    $\begingroup$ While your explanation is correct, your conclusion is not. The expectation does indeed correspond to an averaging over a collection of trajectories since you need to consider all the possible combinations of states and actions that lead to the return $G_t$ and weight them by how probable the combination is under the policy $\pi$ and the transition function $T$. $\endgroup$
    – Diego Gomez
    Apr 4, 2020 at 0:50
  • 1
    $\begingroup$ @DiegoGomez Have added an edit, thanks for pointing out. $\endgroup$
    – ijuneja
    Apr 4, 2020 at 4:34
  • $\begingroup$ Thanks. So in the equation imgur.com/4g6fjIU the flashed symbols are the ones that denote the probabilities of the expectation formula? $\endgroup$
    – Jack
    Apr 6, 2020 at 16:29
  • 1
    $\begingroup$ @Jack in the linked image, summation symbols have been highlighted. The inner summation is over the possible next-state, reward tuple and the outer one is over a stochastic policy, so yes all of them together are capturing the expectation if that is what you meant. $\endgroup$
    – ijuneja
    Apr 6, 2020 at 16:34
  • 1
    $\begingroup$ @Jack yes I agree, perhaps to be rigorous, it should be made explicit that the expectation is over both $\pi$ and the transition distribution $p$. $\endgroup$
    – ijuneja
    Apr 6, 2020 at 16:37
2
$\begingroup$

In addition to this answer, I would like to note that, if the future trajectories were fixed (i.e. the environment and the policies were deterministic, and the agent always starts from the same state), the expectation of the sum (of the fixed rewards) would simply correspond to the actual sum, because the sum is a constant (i.e. the expectation of a constant is the constant itself), so the expectation operator also applies to the deterministic cases. Therefore, the expectation is a general way of expressing the value of a state in all possible cases (both when trajectories are fixed or not).

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .