Hot answers tagged

11 votes
Accepted

Why are lambda returns so rarely used in policy gradients?

That can be done. For example, Chapter 13 of the 2nd edition of Sutton and Barto's Reinforcement Learning book (page 332) has pseudocode for "Actor Critic with Eligibility Traces". It's using $G_t^{\...
user avatar
  • 9,369
9 votes

Why are lambda returns so rarely used in policy gradients?

Recent actor-critic algorithms do use $\lambda$-returns, but they are disguised as something called the Generalized Advantage Estimator defined as $A^{GAE}_t = \sum_{i=0}^{\infty} (\gamma\lambda)^i \...
user avatar
7 votes
Accepted

What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
user avatar
  • 23.8k
6 votes
Accepted

How can the $\lambda$-return be defined recursively?

To rewrite $G_t^\lambda$ recursively, our goal is to define it in terms of $$G_{t+1}^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t+1:t+n+1}.\tag{0}$$ The $\lambda$-return is a weighted ...
user avatar
5 votes

What is a time-step in a Markov Decision Process?

In a Markov Decision Process (MDP) model, we define a set of states ($S$), a set of actions ($A$), the rewards ($R$), and the transition probabilities $P(s' \mid s, a)$. The goal is to figure out the ...
user avatar
  • 365
4 votes

What is a time-step in a Markov Decision Process?

In the reinforcement learning setting, an agent interacts with an environment in (discrete) time steps, which are incremented after the agent takes an action, receives a reward and the "system" (the ...
user avatar
  • 33.7k
3 votes
Accepted

What is the difference between a reward and a value for a given state?

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
user avatar
  • 23.8k
3 votes

What is the difference between a reward and a value for a given state?

What is the difference between a reward and a value for a given state? Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
user avatar
3 votes
Accepted

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
user avatar
  • 33.7k
3 votes
Accepted

Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, ...
user avatar
3 votes
Accepted

Is the expected value we sample in TD-learning action-value Q or state-value V?

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that? MC and TD are methods for associating value estimates to time step based on experienced ...
user avatar
  • 23.8k
3 votes
Accepted

What is wrong with equation 7.3 in Sutton & Barto's book?

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are ...
user avatar
3 votes

Is my interpretation of the return correct?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
user avatar
  • 23.8k
3 votes
Accepted

Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.
user avatar
3 votes
Accepted

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
user avatar
  • 23.8k
3 votes
Accepted

How to evaluate an RL algorithm when used in a game?

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
user avatar
  • 365
2 votes
Accepted

Is my understanding of the value function, Q function, policy, reward and return correct?

Value function: How good it is to be in a state $s$ following policy $\pi$. There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a ...
user avatar
  • 33.7k
2 votes

Is there any difference between reward and return in reinforcement learning?

As the accepted answer states, the return at the current timestep is equal to the sum of discounted rewards from all future timesteps until the end of the episode. In Chapter 5 of Sutton and Barto, ...
user avatar
  • 926
2 votes

How do I calculate the return given the discount factor and a sequence of rewards?

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation. $$ G_0 = R_0 + \gamma G_1 $$ $$ G_0 = 5 + \gamma\sum_{k=0}^\...
user avatar
  • 961
2 votes

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Note that for a general policy $\pi$ we have that $q_{\pi}(s,a) = \mathbb{E}_{\pi}[G_t | S_t = s, A_t = a]$, where in state $S_t$ we take action $a$ and thereafter following policy $\pi$. Note that ...
user avatar
2 votes

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? That is the definition of return. In fact when applying a discount factor this should formally be ...
user avatar
  • 23.8k
2 votes
Accepted

When updating the state-action value in the Monte Carlo method, is the return the same for each state-action pair?

The discussion uses poor notation, there should be a time index. You obtain a list of tuples $(s_t, a_t, r_t, s_{t+1})$ and then, for every visit MC, you update $$Q(s_t, a_t) = Q(s_t, a_t) + \alpha (...
user avatar
1 vote
Accepted

In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward? By the return (sum of all rewards) from the whole episode. A lot of ...
user avatar
  • 23.8k
1 vote

When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and ...
user avatar
1 vote

How do I calculate the return given the discount factor and a sequence of rewards?

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation. I will show how to do it for the generic case here ...
user avatar
  • 23.8k
1 vote
Accepted

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$? This is partly a notation issue, but $G_t$ is already the future sum of rewards as ...
user avatar
  • 23.8k
1 vote

Is my understanding of the value function, Q function, policy, reward and return correct?

I think most of it is correct. Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to ...
user avatar
  • 313
1 vote

What is the difference between return and expected return?

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
user avatar
  • 111
1 vote

What is the difference between return and expected return?

Formally, the return (also known as the cumulative future discounted reward) can be defined as $$ G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1}, $$ where $0 \leq \gamma \leq 1$ is the discount factor and ...
user avatar
  • 33.7k
1 vote
Accepted

Given specific rewards, how can I calculate the returns for each time step?

Perfect. To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11). $$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$ You'll ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible