# Questions tagged [return]

For questions related to the concept of return in reinforcement learning, which is defined as the future cumulative (discounted) reward or, in simple words, the reward in the long run.

25 questions
Filter by
Sorted by
Tagged with
39 views

### In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

I am trying to understand the code mechanics when selecting the elite states and elite actions. It appears clear to me that they are those that appear in the episodes with the rewards bigger than the ...
43 views

### How do I represent sample efficiency of RL rewards in mathematical notation?

I define sample efficiency as the area under the curve/graph, where $x$-axis is the number of episodes while y-axis is the cumulative reward for that episode. I would like to formally define it with a ...
162 views

### What is the difference between a reward and a value for a given state?

I am trying to learn reinforcement learning and I am focusing on the value iteration. I am looking at the example of grid world, and I am trying to implement it in python. While doing this, I ...
36 views

### How to formulate discounted return in cartpole?

I am trying to formulate a problem that aims to prolong the lifetime of the simulation, the same as the Cartpole problem. I aware that there are two types of return: finite horizon undiscounted ...
58 views

### When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

Referring to this post, in the following formula to update the state-action value $$Q(s,a) = Q(s,a) + \alpha (G − Q(s,a)),$$ is the value of $G$ (the return) the same for every state-action $(s,a)$ ...
46 views

### For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$? In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
42 views

### Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$? The return for an MDP is usually defined as $$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$ Why is this ...
114 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
88 views

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
106 views

125 views

### Given specific rewards, how can I calculate the returns for each time step?

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
931 views

### Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
72 views

### Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?
I've seen numerous mathematical explanations of reward, value functions $V(s)$, and return functions. The reward provides an immediate return for being in a specific state. The better the reward, the ...
The "discounted sum of future rewards" (or return) using discount factor $\gamma$ is $$\gamma^1 r_1 +\gamma^2 r_2 + \gamma^3 r_2 + \dots \tag{1}\label{1}$$ where $r_i$ is the reward received ...