7
votes
Accepted
Is the discount not needed in a deterministic environment for Reinforcement Learning?
The motivation for adding the discount factor $\gamma$ is generally, at least initially, based simply in "theoretical convenience". Ideally, we'd like to define the "objective" of an RL agent as ...
- 9,419
5
votes
Accepted
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
This question is really getting at the meaning of the discount factor in Markov decision processes. There are actually two, equivalent ways of interpreting the discount factor.
The first is probably ...
- 931
3
votes
Accepted
For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
$T = \infty$ and $\gamma = 1$ cannot be both true at the same time because the return defined in equation 3.11 is supposed to be a unified definition of the return for both continuing and episodic ...
- 34.3k
3
votes
Accepted
What should the discount factor for the non-slippery version of the FrozenLake environment be?
After trying to understand what was happening by going through the algorithm on paper (below), I found all values were the same, so greedifying with respect to the value function often just had the ...
- 24.5k
3
votes
What is the value of a state when there is a certain probability that agent will die after each step?
The value of a state depends on the policy that you use, so I'll make the assumption here that you're talking about value using the optimal policy.
According to the optimal policy, the agent would ...
- 66
2
votes
Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?
The discount factor is not something you should be optimising. It is typically part of the problem statement.
For practical purposes, you may set it below 1.0 for continuous problems when in fact you ...
- 24.5k
2
votes
How do I calculate the return given the discount factor and a sequence of rewards?
You know all the rewards. They're 5, 7, 7, 7, and 7s forever. The problem now boils down to essentially a geometric series computation.
$$
G_0 = R_0 + \gamma G_1
$$
$$
G_0 = 5 + \gamma\sum_{k=0}^\...
- 981
2
votes
Accepted
What is the value of a state when there is a certain probability that agent will die after each step?
I will fill in some details in shaabhishek's answer for people who are interested.
With this in mind, what is the value of a square (1,1)?
First of all, the value function is dependent on a policy....
- 938
1
vote
How do I calculate the return given the discount factor and a sequence of rewards?
There are a few ways to resolve values of infinite sums. In this case, we can use a simple technique of self-reference to create a solvable equation.
I will show how to do it for the generic case here ...
- 24.5k
1
vote
Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning
Personally, I find the best way to think of SMDPs intuitively by just imagining that you just discretise the time into such small steps (infinitesimally small steps if necessary) that you can treat it ...
- 9,419
1
vote
When discounted MAB is useful?
One of the reasons a discount factor is used, is to make sure the reward maximization is a well-defined problem and to make the sum of all rewards convergent.
In the MAB problem, the number of trials ...
- 111
1
vote
Accepted
In online one step actor critic, why does the weights update become less significant as the episode progresses?
This "decay" of later values is a direct consequence of the episodic formula for the objective function for REINFORCE:
$$J(\theta) = v_{\pi_\theta}(s_0)$$
That is, the expected return from the first ...
- 24.5k
1
vote
Accepted
Are there any discount-factors based on branching factors?
First, an important note on any form of discounting: adding a discount factor can change what the optimal policy is. The optimal policy when a discount factor is present can be different from the ...
- 9,419
Only top scored, non community-wiki answers of a minimum length are eligible