Questions tagged [discount-factor]
Use for questions involving discount factor(s) in reinforcement learning.
26
questions
1
vote
0
answers
38
views
What is the definition of Q when the discount factor depends on time?
Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by,
$$Q(a,s)=E_{\pi}\big[...
- 111
0
votes
0
answers
159
views
Convergence of Value Iteration for Discount factor of 1
Given this pseudo code for value iteration:
In the case of gamma=1, under what conditions on the MDP will we still be able to find the optimal policy?
1
vote
1
answer
185
views
Effects of hyperparameters in Q-learning
While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself.
Firstly, I noticed that increasing the ...
1
vote
1
answer
184
views
PPO: dealing with variable episodic length
I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
0
votes
1
answer
132
views
Should the concept of discounted rewards result in multiple arrays per episode in RL?
Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below:
Say you have a reward array of length <...
- 89
3
votes
0
answers
132
views
What is the correct interpretation of the discount factor in MDPs?
In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
- 31
0
votes
0
answers
67
views
Policy and Discount Factor
This question is similar to this question, however it has a different question.
I'm learning MDP's and I'd like to know if I'm doing these exercises correctly:
Consider the following MDP:
Suppose a ...
- 145
4
votes
1
answer
411
views
What should the discount factor for the non-slippery version of the FrozenLake environment be?
I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
1
vote
0
answers
34
views
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):
Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
- 179
0
votes
2
answers
335
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 1,171
2
votes
1
answer
924
views
How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?
I am trying to use reinforcement learning to solve a task and compare its performance to humans.
The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
- 203
2
votes
0
answers
113
views
Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?
I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as a Markov Decision Process with a non-zero discount ...
- 563
6
votes
1
answer
257
views
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being
\begin{equation}
\mu(s) = \frac{\...
- 103
2
votes
1
answer
153
views
Relation between discounted MDP and stochastic shortest path problems in RL
I have been reading about discounted MDPs and Stochastic Shortest Path (SSP). I recently came to know (from a friend) that every discounted MDP can be converted to an equivalent SSP but not the other ...
- 359
4
votes
1
answer
691
views
Why do we discount the state distribution?
In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns.
I have also seen mention of discounted state ...
- 4,600
1
vote
1
answer
122
views
For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?
For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$?
In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
- 717
2
votes
0
answers
151
views
Combine DQN with the Average Reward setting
I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
- 919
2
votes
2
answers
2k
views
How do I calculate the return given the discount factor and a sequence of rewards?
I know that $G_t = R_{t+1} + G_{t+1}$.
Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$?
As it's infinite, how can we ...
- 21
2
votes
1
answer
224
views
Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning
Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an ...
- 471
3
votes
2
answers
232
views
What is the value of a state when there is a certain probability that agent will die after each step?
We assume infinite horizon and discount factor $\gamma = 1$. At each step, after the agent takes an action and gets its reward, there is a probability $\alpha = 0.2$, that agent will die. The assumed ...
1
vote
1
answer
177
views
When discounted MAB is useful?
Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing ...
- 471
4
votes
0
answers
62
views
Does everyone still use discount rates?
In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
- 2,028
2
votes
1
answer
1k
views
In online one step actor critic, why does the weights update become less significant as the episode progresses?
The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm.
Why do the weights updates depend on the discount factor via $I$?
It seems that the more ...
- 729
4
votes
2
answers
147
views
Are there any discount-factors based on branching factors?
I recently came across this function:
$$\sum_{t = 0}^{\infty} \gamma^t R_t.$$
It's elegant and looks to be useful in the type of deterministic, perfect-information, finite models I'm working with.
...
- 6,237
5
votes
1
answer
749
views
Is the discount not needed in a deterministic environment for Reinforcement Learning?
I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero:
Self-play
In AlphaGo Zero, the NN is used to ...
- 1,027
4
votes
1
answer
2k
views
Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?
I'm now reading a book titled as Hands-On Reinforcement Learning with Python, and the author explains the discount factor that is used in Reinforcement Learing to discount the future reward, with the ...
- 1,027