11
votes
Accepted
Why are lambda returns so rarely used in policy gradients?
That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
- 10.1k
9
votes
Why are lambda returns so rarely used in policy gradients?
Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
- 226
7
votes
Accepted
What is the difference between expected return and value function?
There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
- 30.5k
6
votes
Accepted
Is there any difference between reward and return in reinforcement learning?
Return refers to the total discounted reward, starting from the current timestep.
- 1,131
6
votes
Accepted
How can the $\lambda$-return be defined recursively?
To rewrite $G_t^\lambda$ recursively, our goal is to define it in terms of
$$G_{t+1}^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t+1:t+n+1}.\tag{0}$$
The $\lambda$-return is a weighted ...
- 2,038
5
votes
What is a time-step in a Markov Decision Process?
In a Markov Decision Process (MDP) model, we define a set of states ($S$), a set of actions ($A$), the rewards ($R$), and the transition probabilities $P(s' \mid s, a)$. The goal is to figure out the ...
- 365
4
votes
What is a time-step in a Markov Decision Process?
In the reinforcement learning setting, an agent interacts with an environment in (discrete) time steps, which are incremented after the agent takes an action, receives a reward and the "system" (the ...
- 39.7k
4
votes
Accepted
For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
- 39.7k
4
votes
Accepted
What is the difference between a reward and a value for a given state?
Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
- 30.5k
3
votes
What is the difference between a reward and a value for a given state?
What is the difference between a reward and a value for a given state?
Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
3
votes
Accepted
Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?
It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, ...
- 4,685
3
votes
Accepted
Is the expected value we sample in TD-learning action-value Q or state-value V?
However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that?
MC and TD are methods for associating value estimates to time step based on experienced ...
- 30.5k
3
votes
Accepted
What is wrong with equation 7.3 in Sutton & Barto's book?
In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are ...
- 4,685
3
votes
Is my interpretation of the return correct?
Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
- 30.5k
3
votes
Accepted
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$?
The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
- 30.5k
3
votes
Is there any difference between reward and return in reinforcement learning?
As the accepted answer states, the return at the current timestep is equal to the sum of discounted rewards from all future timesteps until the end of the episode. In Chapter 5 of Sutton and Barto, ...
- 1,332
3
votes
Accepted
How to evaluate an RL algorithm when used in a game?
When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
- 365
2
votes
Accepted
Is my understanding of the value function, Q function, policy, reward and return correct?
Value function: How good it is to be in a state $s$ following policy $\pi$.
There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a ...
- 39.7k
2
votes
How do I calculate the return given the discount factor and a sequence of rewards?
You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.
$$
G_0 = R_0 + \gamma G_1
$$
$$
G_0 = 5 + \gamma\sum_{k=0}^\...
- 1,126
2
votes
Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?
Note that for a general policy $\pi$ we have that $q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]$, where in state $S_t$ we take action $a$ and thereafter following policy $\pi$. Note that ...
- 4,685
2
votes
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?
That is the definition of return.
In fact when applying a discount factor this should formally be ...
- 30.5k
2
votes
Accepted
When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?
The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s_t, a_t, r_t, s_{t+1})$ and then, for every visit MC, you update
$$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (...
- 4,685
1
vote
Accepted
In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?
My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward?
By the return (sum of all rewards) from the whole episode. A lot of ...
- 30.5k
1
vote
How do I represent sample efficiency of RL rewards in mathematical notation?
Episodes are discrete, there is no need for calculus. Your "sample efficiency" metric is:
$$\sum_{x=a}^b R_x$$
The quantity you are measuring per episode is the return (undiscounted). The ...
- 30.5k
1
vote
When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and ...
1
vote
How do I calculate the return given the discount factor and a sequence of rewards?
There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.
I will show how to do it for the generic case here ...
- 30.5k
1
vote
Accepted
Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?
shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$?
This is partly a notation issue, but $G_t$ is already the future sum of rewards as ...
- 30.5k
1
vote
Is my understanding of the value function, Q function, policy, reward and return correct?
I think most of it is correct.
Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to ...
- 333
1
vote
What is the difference between return and expected return?
You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
- 111
1
vote
What is the difference between return and expected return?
Formally, the return (also known as the cumulative future discounted reward) can be defined as
$$
G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},
$$
where $0 \leq \gamma \leq 1$ is the discount factor and ...
- 39.7k
Only top scored, non community-wiki answers of a minimum length are eligible