# Tag Info

Accepted

• 226
Accepted

### What is the difference between expected return and value function?

There is a strong relationship between a value function and a return. Namely that a value function calculates the expected return from being in a certain state, or taking a specific action in a ...
• 23.8k
Accepted

### How can the $\lambda$-return be defined recursively?

To rewrite $G_t^\lambda$ recursively, our goal is to define it in terms of $$G_{t+1}^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t+1:t+n+1}.\tag{0}$$ The $\lambda$-return is a weighted ...
• 1,948

### What is a time-step in a Markov Decision Process?

In a Markov Decision Process (MDP) model, we define a set of states ($S$), a set of actions ($A$), the rewards ($R$), and the transition probabilities $P(s' \mid s, a)$. The goal is to figure out the ...
• 365

### What is a time-step in a Markov Decision Process?

In the reinforcement learning setting, an agent interacts with an environment in (discrete) time steps, which are incremented after the agent takes an action, receives a reward and the "system" (the ...
• 33.7k
Accepted

### What is the difference between a reward and a value for a given state?

Starting with rewards, states don't have rewards in general. A reward is a number returned at a certain step of the MDP. If you arrange things in sequence over a whole time step $s, a, r, s'$ for ...
• 23.8k

### What is the difference between a reward and a value for a given state?

What is the difference between a reward and a value for a given state? Let us say that an agent took an action from state $A$ and reached state $B$ and got a score $R$. This instantaneous score the ...
Accepted

### For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
• 33.7k
Accepted

### Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, ...
• 4,055
Accepted

### Is the expected value we sample in TD-learning action-value Q or state-value V?

However, from the blogs and texts I read, the equations are expressed in terms of V and NOT Q. Why is that? MC and TD are methods for associating value estimates to time step based on experienced ...
• 23.8k
Accepted

### What is wrong with equation 7.3 in Sutton & Barto's book?

In general, $\mathbb{E}_\pi[G_{t:t+n}|S_t = s] \neq v_\pi(s)$. $v_\pi(s)$ is defined as $\mathbb{E}_\pi[\sum_{k=0}^{\infty} \gamma^k R_{t+k+1} | S_t = s]$, so you should be able to see why the two are ...
• 4,055

### Is my interpretation of the return correct?

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of ...
• 23.8k
Accepted

### Is there any difference between reward and return in reinforcement learning?

Return refers to the total discounted reward, starting from the current timestep.
Accepted

### Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

Can someone provide the reasoning behind why $G_{t+1}$ is equal to $v_*(S_{t+1})$? The two things are not usually exactly equal, because $G_{t+1}$ is a probability distribution over all possible ...
• 23.8k
Accepted

### How to evaluate an RL algorithm when used in a game?

When you want to compare Reinforcement Learning algorithms, you might want to compare the average rewards they generate and how fast and close they get to the optimal policy. However, in the case of ...
• 365
Accepted

### Is my understanding of the value function, Q function, policy, reward and return correct?

Value function: How good it is to be in a state $s$ following policy $\pi$. There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a ...
• 33.7k

### Is there any difference between reward and return in reinforcement learning?

As the accepted answer states, the return at the current timestep is equal to the sum of discounted rewards from all future timesteps until the end of the episode. In Chapter 5 of Sutton and Barto, ...
• 926

• 4,055
1 vote
Accepted

### In the cross-entropy method, should I select state-action pairs by their immediate reward or by the episode reward?

My question is if I should select state_action pairs by theirs immediate reward or should I select them by the episode reward? By the return (sum of all rewards) from the whole episode. A lot of ...
• 23.8k
1 vote

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and ...
1 vote

### How do I calculate the return given the discount factor and a sequence of rewards?

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation. I will show how to do it for the generic case here ...
• 23.8k
1 vote
Accepted

### Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

shouldn't the expected return be calculated for some faraway time in the future (𝑡+𝑛) instead of current time $t$? This is partly a notation issue, but $G_t$ is already the future sum of rewards as ...
• 23.8k
1 vote

### Is my understanding of the value function, Q function, policy, reward and return correct?

I think most of it is correct. Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to ...
• 313
1 vote

### What is the difference between return and expected return?

You're correct, the return is the discounted future reward from the one iteration while the expected return is averaged over a bunch of iterations.
• 111
1 vote

### What is the difference between return and expected return?

Formally, the return (also known as the cumulative future discounted reward) can be defined as $$G_t = \sum_{k=0}^\infty \gamma^k R_{t+k+1},$$ where $0 \leq \gamma \leq 1$ is the discount factor and ...
• 33.7k
1 vote
Accepted

### Given specific rewards, how can I calculate the returns for each time step?

Perfect. To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11). $$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$ You'll ...
• 1,948

Only top scored, non community-wiki answers of a minimum length are eligible