3
$\begingroup$

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$.

If an agent is following a policy (which in itself is a probability distribution of choosing a next state $S_{t+1}$ from $S_{t}$), the agent wants to find the value at $S_{t}$ by calculating sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

Is my understanding correct?

$\endgroup$
0
$\begingroup$

Formally, the return (in this case, also called cumulative future discounted reward) can be defined as

$$ R_t = \sum_{k=0}^\infty \gamma^k r_{t+k+1}, $$

where $\gamma$ is the discount factor and $r_{i}$ is the reward at time step $i$, and the expected return is defined as

\begin{align} v^\pi(s) &= \mathbb{E}\left[R_t \mid s_t = s \right] \\ &= \mathbb{E}\left[\sum_{k=0}^\infty \gamma^k r_{t+k+1} \bigm\vert s_t = s \right] \end{align}

In other words, the value of a state $s$ (associated with a policy $\pi$) is equal to the expectation of the return given that $s_t = s$.

A policy is not a probability distribution of choosing the next state. A stochastic policy is a family of a conditional probability distribution over actions given states. There are also deterministic policies. Have a look at this question What is the difference between a stochastic and a deterministic policy? for more details about the definition of stochastic and deterministic policies.

If an agent is following a policy, the agent wants to find the value at $S_{t}$ by calculating sort of "weighted average" of all the returns from $S_{t}.$ This is called the expected return.

In the case of Monte Carlo Prediction, the value of a state associated with a specific policy, that is, the expected value of the return given a state is approximated with a finite (weighted) average. See e.g. What is the difference between First-Visit Monte-Carlo and Every-Visit Monte-Carlo Policy Evaluation?. Furthermore, note that the expectation of a discrete random variable is defined as a weighted average. However, the return is not a discrete random variable, but, in general, it is a continuous one.

$\endgroup$
  • $\begingroup$ Isn't returns and actual returns the same terminology? $\endgroup$ – DuttaA Jun 30 at 15:56
  • $\begingroup$ @DuttaA What do you mean by "actual returns"? $\endgroup$ – nbro Jun 30 at 15:57
  • $\begingroup$ Return after one iteration of MC visit or maybe TD or the best dynamic way. $\endgroup$ – DuttaA Jun 30 at 16:12
  • $\begingroup$ @DuttaA I don't understand what you're asking. $\endgroup$ – nbro Jun 30 at 16:20
  • 2
    $\begingroup$ It would help the OP I think if you more carefully differentiate and explain the difference between random distributions (usually noted with capital letters e.g. $R_t$) and data/observations (usually noted with lower-case letters e.g. $r_t$). The notation you are currently using in this answer is very loose in that regard $\endgroup$ – Neil Slater Jun 30 at 16:33
0
$\begingroup$

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.