7 votes
Accepted

Is the discount not needed in a deterministic environment for Reinforcement Learning?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
user avatar
  • 9,419
5 votes
Accepted

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor. The first is probably ...
user avatar
  • 931
3 votes
Accepted

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
user avatar
  • 34.3k
3 votes
Accepted

What should the discount factor for the non-slippery version of the FrozenLake environment be?

After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
user avatar
  • 24.5k
3 votes

What is the value of a state when there is a certain probability that agent will die after each step?

The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy. According to the optimal policy, the agent would ...
user avatar
2 votes

Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?

The discount factor is not something you should be optimising. It is typically part of the problem statement. For practical purposes, you may set it below 1.0 for continuous problems when in fact you ...
user avatar
  • 24.5k
2 votes

How do I calculate the return given the discount factor and a sequence of rewards?

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation. $$ G_0 = R_0 + \gamma G_1 $$ $$ G_0 = 5 + \gamma\sum_{k=0}^\...
user avatar
  • 981
2 votes
Accepted

What is the value of a state when there is a certain probability that agent will die after each step?

I will fill in some details in shaabhishek's answer for people who are interested. With this in mind, what is the value of a square (1,1)? First of all, the value function is dependent on a policy....
user avatar
  • 938
1 vote

How do I calculate the return given the discount factor and a sequence of rewards?

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation. I will show how to do it for the generic case here ...
user avatar
  • 24.5k
1 vote

Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it ...
user avatar
  • 9,419
1 vote

When discounted MAB is useful?

One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent. In the MAB problem, the number of trials ...
user avatar
1 vote
Accepted

In online one step actor critic, why does the weights update become less significant as the episode progresses?

This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE: $$J(\theta) = v_{\pi_\theta}(s_0)$$ That is, the expected return from the first ...
user avatar
  • 24.5k
1 vote
Accepted

Are there any discount-factors based on branching factors?

First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the ...
user avatar
  • 9,419

Only top scored, non community-wiki answers of a minimum length are eligible