Questions tagged [discount-factor]

Use for questions involving discount factor(s) in reinforcement learning.

Filter by
Sorted by
Tagged with
0 votes
1 answer
21 views

Relationship between regularization and (effective) discounting in deep Q learning

I have a deep-Q-network-type reinforcement learner in a minigrid-type environment. After learning I can put the agent in a series of contrived situations and measure its Q values, and then infer its ...
dinnums's user avatar
1 vote
0 answers
40 views

What is the definition of Q when the discount factor depends on time?

Suppose I want to find out the Q value of a particular state $s$ bu doing action $a$ at a particular timestep $t$. I know that the Q-value when the discount factor is given by, $$Q(a,s)=E_{\pi}\big[...
Asher2211's user avatar
  • 111
0 votes
0 answers
296 views

Convergence of Value Iteration for Discount factor of 1

Given this pseudo code for value iteration: In the case of gamma=1, under what conditions on the MDP will we still be able to find the optimal policy?
Toffe1369's user avatar
1 vote
1 answer
247 views

Effects of hyperparameters in Q-learning

While playing around with the learning rate and discount factor in the Q-learning algorithm, I noticed some behavior that I could not really understand myself. Firstly, I noticed that increasing the ...
perceptronEnthusiast420's user avatar
1 vote
1 answer
245 views

PPO: dealing with variable episodic length

I'm dealing with a project that has episodes of variable length raging from just 3 steps to 20 steps. Now, I'm guessing that this may cause problems with GAE, as actions in large episodes will have ...
Antonis Karvelas's user avatar
0 votes
1 answer
168 views

Should the concept of discounted rewards result in multiple arrays per episode in RL?

Note that I'm coming from mostly only working with the REINFORCE algorithm, but I've typically seen discounted rewards calculated in a way that looks like below: Say you have a reward array of length <...
Josh's user avatar
  • 89
4 votes
0 answers
171 views

What is the correct interpretation of the discount factor in MDPs?

In infinite-horizon MDPs one can consider the expected discounted return from a distribution of start states as the objective[^1]. i.e. $\mathbb{E}[V^{\pi}(S_0)] = \mathbb{E}[G_0] = \mathbb{E}[\sum_{t=...
Skander's user avatar
  • 41
0 votes
0 answers
77 views

Policy and Discount Factor

This question is similar to this question, however it has a different question. I'm learning MDP's and I'd like to know if I'm doing these exercises correctly: Consider the following MDP: Suppose a ...
user's user avatar
  • 145
4 votes
1 answer
537 views

What should the discount factor for the non-slippery version of the FrozenLake environment be?

I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
ketan dhanuka's user avatar
1 vote
0 answers
44 views

Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?

Sutton-Barto (Section 10.4, page 254): Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
user3489173's user avatar
0 votes
2 answers
392 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
Taw's user avatar
  • 1,241
2 votes
1 answer
1k views

How to encourage the reinforcement-learning agent to reach the goal as quickly as possible, and what's the effect of discount factor?

I am trying to use reinforcement learning to solve a task and compare its performance to humans. The task is to find a single target in a fixed number of locations. At each step, the agent will pick ...
Cloudy's user avatar
  • 213
2 votes
0 answers
124 views

Is it better to model a Contextual Multi-Armed Bandit problem as an MDP with a non-zero discount factor than treating it as it is?

I'd like to ask if it is, generally, better to model a problem that naturally appears as a Contextual Multi-Armed Bandit like Recommender Systems as a Markov Decision Process with a non-zero discount ...
Daviiid's user avatar
  • 573
6 votes
1 answer
286 views

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{equation} \mu(s) = \frac{\...
Felipe Costa's user avatar
2 votes
2 answers
197 views

Relation between discounted MDP and stochastic shortest path problems in RL

I have been reading about discounted MDPs and Stochastic Shortest Path (SSP). I recently came to know (from a friend) that every discounted MDP can be converted to an equivalent SSP but not the other ...
user529295's user avatar
4 votes
2 answers
899 views

Why do we discount the state distribution?

In Reinforcement Learning, it is common to use a discount factor $\gamma$ to give less importance to future rewards when calculating the returns. I have also seen mention of discounted state ...
David's user avatar
  • 4,770
2 votes
1 answer
148 views

For episodic tasks with an absorbing state, why can't we both have $\gamma=1$ and $T= \infty$ in the definition of the return?

For episodic tasks with an absorbing state, why can't $\gamma=1$ and $T= \infty$? In Sutton and Barto's book, they say that, for episodic tasks with absorbing states that becomes an infinite sequence, ...
user8714896's user avatar
2 votes
0 answers
171 views

Combine DQN with the Average Reward setting

I have to deal with a non-episodic task, where there is addittionally a continuous state space and more specifically in each time step there is always a new state that has never been seen before. I ...
ddaedalus's user avatar
  • 919
2 votes
2 answers
2k views

How do I calculate the return given the discount factor and a sequence of rewards?

I know that $G_t = R_{t+1} + G_{t+1}$. Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$? As it's infinite, how can we ...
Thom's user avatar
  • 21
2 votes
1 answer
243 views

Updating action-value functions in Semi-Markov Decision Process and Reinforcement Learning

Suppose that the transition time between two states is a random variable (for example, unknown exponential distribution); and between two arrivals, there is no reward. If $\tau$ (real number not an ...
Amin's user avatar
  • 481
3 votes
2 answers
254 views

What is the value of a state when there is a certain probability that agent will die after each step?

We assume infinite horizon and discount factor $\gamma = 1$. At each step, after the agent takes an action and gets its reward, there is a probability $\alpha = 0.2$, that agent will die. The assumed ...
Milan Mitterko's user avatar
1 vote
1 answer
188 views

When discounted MAB is useful?

Many of multi-armed bandit(MAB) algorithms are used when the total reward is the sum of all rewards. However, in RL, the discounted reward is mainly used. Why is the discounted reward not prevailing ...
Amin's user avatar
  • 481
4 votes
0 answers
64 views

Does everyone still use discount rates?

In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
Philip Raeisghasem's user avatar
2 votes
1 answer
1k views

In online one step actor critic, why does the weights update become less significant as the episode progresses?

The Reinforcement Learning Book by Richard Sutton et al, section 13.5 shows an online actor critic algorithm. Why do the weights updates depend on the discount factor via $I$? It seems that the more ...
Gulzar's user avatar
  • 749
4 votes
2 answers
147 views

Are there any discount-factors based on branching factors?

I recently came across this function: $$\sum_{t = 0}^{\infty} \gamma^t R_t.$$ It's elegant and looks to be useful in the type of deterministic, perfect-information, finite models I'm working with. ...
DukeZhou's user avatar
  • 6,237
5 votes
1 answer
778 views

Is the discount not needed in a deterministic environment for Reinforcement Learning?

I'm now reading a book titled as "Deep Reinforcement Learning Hands-On" and the author said the following on the chapter about AlphaGo Zero: Self-play In AlphaGo Zero, the NN is used to ...
Blaszard's user avatar
  • 1,037
4 votes
1 answer
2k views

Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?

I'm now reading a book titled as Hands-On Reinforcement Learning with Python, and the author explains the discount factor that is used in Reinforcement Learing to discount the future reward, with the ...
Blaszard's user avatar
  • 1,037