# Questions tagged [return]

For questions related to the concept of return in reinforcement learning, which is defined as the future cumulative (discounted) reward or, in simple words, the reward in the long run.

25 questions
Filter by
Sorted by
Tagged with
1 vote
45 views

### In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
105 views

### How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
1 vote
420 views

### What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
55 views

### How to formulate discounted return in cartpole?

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ...
1 vote
78 views

### When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

Referring to this post, in the following formula to update the state-action value $$Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$ is the value of $G$ (the return) the same for every state-action $(s,a)$ ...
1 vote
57 views

### For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$? In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
84 views

### Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$? The return for an MDP is usually defined as $$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$ Why is this ...
132 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
103 views

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
127 views

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)|$$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}... 2 votes 1 answer 1k views ### Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute \mathbb{E}(R \mid s, a) (the expected return for taking ... 1 vote 2 answers 676 views ### How do I calculate the return given the discount factor and a sequence of rewards? I know that G_t = R_{t+1} + G_{t+1}. Suppose \gamma = 0.9 and the reward sequence is R_1 = 2 followed by an infinite sequence of 7s. What is the value of G_0? As it's infinite, how can we ... 1 vote 0 answers 59 views ### Why does the n-step return being zero result in high variance in off policy n-step TD? In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book: G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T where G_{h:h} = V_{h-1}(S_h). (Recall that this return is used at ... 4 votes 2 answers 123 views ### Why is G_{t+1} is replaced with v_*(S_{t+1}) in the Bellman optimality equation? In equation 3.17 of Sutton and Barto's book:$$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$G_{t+1} here have been replaced with v_*(S_{t+1}), but no reason has ... 5 votes 2 answers 1k views ### Is there any difference between reward and return in reinforcement learning? I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ... 3 votes 1 answer 46 views ### Shouldn't expected return be calculated for some faraway time in the future t+n instead of current time t? I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ... 2 votes 2 answers 448 views ### Is my understanding of the value function, Q function, policy, reward and return correct? I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy π. ... 4 votes 1 answer 325 views ### How to evaluate an RL algorithm when used in a game? I'm planning to create a web-based RL board game, and I wondered how I would evaluate the performance of the RL agent. How would I be able to say, "Version X performed better than version Y, as ... 4 votes 2 answers 280 views ### What is the difference between return and expected return? At a time step t, for a state S_{t}, the return is defined as the discounted cumulative reward from that time step t. If an agent is following a policy (which in itself is a probability ... 2 votes 1 answer 820 views ### How can the \lambda-return be defined recursively? The \lambda-return is defined as$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$where$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...