Skip to main content
7 votes
Accepted

Is the discount not needed in a deterministic environment for Reinforcement Learning?

The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
Dennis Soemers's user avatar
  • 10.4k
5 votes
Accepted

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor. The first is probably ...
Taw's user avatar
  • 1,281
4 votes
Accepted

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
nbro's user avatar
  • 40.9k
3 votes
Accepted

What should the discount factor for the non-slippery version of the FrozenLake environment be?

After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
Neil Slater's user avatar
  • 32.7k
3 votes

What is the value of a state when there is a certain probability that agent will die after each step?

The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy. According to the optimal policy, the agent would ...
shaabhishek's user avatar
2 votes
Accepted

Should the concept of discounted rewards result in multiple arrays per episode in RL?

The discount factor is not used to take into account uncertainty, but to encourage the agent to have a longer term view when it takes an action. If the discount factor is close to 0, the agent is ...
Raphael Lopez Kaufman's user avatar
2 votes

Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?

The discount factor is not something you should be optimising. It is typically part of the problem statement. For practical purposes, you may set it below 1.0 for continuous problems when in fact you ...
Neil Slater's user avatar
  • 32.7k
2 votes

How do I calculate the return given the discount factor and a sequence of rewards?

You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation. $$ G_0 = R_0 + \gamma G_1 $$ $$ G_0 = 5 + \gamma\sum_{k=0}^\...
harwiltz's user avatar
  • 1,136
2 votes
Accepted

What is the value of a state when there is a certain probability that agent will die after each step?

I will fill in some details in shaabhishek's answer for people who are interested. With this in mind, what is the value of a square (1,1)? First of all, the value function is dependent on a policy....
DeepQZero's user avatar
  • 1,424
1 vote
Accepted

In a known finite episodic task, is there still value to discount?

Your overall reasoning sounds reasonable. In a known finite episodic task, while the choice of $\gamma$ could be seen as a global hyperparameter as you narrated, the impact of $\gamma$ is more ...
cinch's user avatar
  • 2,277
1 vote

Effects of hyperparameters in Q-learning

Discount factor in (tabular) RL including Q-learning generally acts as a regularization hyperparameter to trade-off optimality with sample efficiency especially for continuous tasks with infinite time ...
cinch's user avatar
  • 2,277
1 vote

Relation between discounted MDP and stochastic shortest path problems in RL

This is discussed in [1], where the authors provide the following proposition: If $FH$ is the class of finite-horizon MDPs, $IFH$ the class of infinite-horizon MDPs and $SSP$ are stochastic shortest-...
BoZenKhaa's user avatar
  • 113
1 vote
Accepted

Why do we discount the state distribution?

Not an exhaustive answer, but perhaps this blog post by Alessio Russo may be helpful. In particular, he states how There is an equivalence between using a discount factor and reaching a terminal ...
Gabriele's user avatar
1 vote

How do I calculate the return given the discount factor and a sequence of rewards?

There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation. I will show how to do it for the generic case here ...
Neil Slater's user avatar
  • 32.7k
1 vote

Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it ...
Dennis Soemers's user avatar
  • 10.4k
1 vote

When discounted MAB is useful?

One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent. In the MAB problem, the number of trials ...
psiyumm's user avatar
  • 111
1 vote
Accepted

In online one step actor critic, why does the weights update become less significant as the episode progresses?

This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE: $$J(\theta) = v_{\pi_\theta}(s_0)$$ That is, the expected return from the first ...
Neil Slater's user avatar
  • 32.7k
1 vote
Accepted

Are there any discount-factors based on branching factors?

First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the ...
Dennis Soemers's user avatar
  • 10.4k

Only top scored, non community-wiki answers of a minimum length are eligible