# What is the difference between return and expected return?

At a time step $$t$$, for a state $$S_{t}$$, the return is defined as the discounted cumulative reward from that time step $$t$$.

If an agent is following a policy (which in itself is a probability distribution of choosing a next state $$S_{t+1}$$ from $$S_{t}$$), the agent wants to find the value at $$S_{t}$$ by calculating sort of "weighted average" of all the returns from $$S_{t}.$$ This is called the expected return.

Is my understanding correct?

Formally, the return (in this case, also called cumulative future discounted reward) can be defined as

$$R_t = \sum_{k=0}^\infty \gamma^k r_{t+k+1},$$

where $$\gamma$$ is the discount factor and $$r_{i}$$ is the reward at time step $$i$$, and the expected return is defined as

\begin{align} v^\pi(s) &= \mathbb{E}\left[R_t \mid s_t = s \right] \\ &= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k r_{t+k+1} \bigm\vert s_t = s \right] \end{align}

In other words, the value of a state $$s$$ (associated with a policy $$\pi$$) is equal to the expectation of the return given that $$s_t = s$$.

A policy is not a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution over actions given states. There are also deterministic policies. Have a look at this question What is the difference between a stochastic and a deterministic policy? for more details about the definition of stochastic and deterministic policies.

If an agent is following a policy, the agent wants to find the value at $$S_{t}$$ by calculating sort of "weighted average" of all the returns from $$S_{t}.$$ This is called the expected return.

In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?. Furthermore, note that the expectation of a discrete random variable is defined as a weighted average. However, the return is not a discrete random variable, but, in general, it is a continuous one.

• Isn't returns and actual returns the same terminology? – DuttaA Jun 30 at 15:56
• @DuttaA What do you mean by "actual returns"? – nbro Jun 30 at 15:57
• Return after one iteration of MC visit or maybe TD or the best dynamic way. – DuttaA Jun 30 at 16:12
• @DuttaA I don't understand what you're asking. – nbro Jun 30 at 16:20
• It would help the OP I think if you more carefully differentiate and explain the difference between random distributions (usually noted with capital letters e.g. $R_t$) and data/observations (usually noted with lower-case letters e.g. $r_t$). The notation you are currently using in this answer is very loose in that regard – Neil Slater Jun 30 at 16:33

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.