Can the optimal value of discount factor in Deep Reinforcement Learning be between 0.2 to 0.8?

Question

I'm now reading a book titled as Hands-On Reinforcement Learning with Python, and the author explains the discount factor that is used in Reinforcement Learing to discount the future reward, with the following:

A discount factor of 0 will never learn considering only the immediate rewards; similarly, a discount factor of 1 will learn forever looking for the future reward, which may lead to infinity. So the optimal value of the discount factor lies between 0.2 to 0.8.

The author seems to be not going to explain further about the figure, but all the tutorials and explanations I have ever read write the optimal (or at least widely used) discount factor between 0.9 an 0.99. This is the first time I have seen such a low-figure discount factor.

All the other explanations the author makes regarding the discount factor are the same as I have read so far.

Is the author correct here or does it depend on cases? If it is, then what kind of problems and/or situations should I set the discount factor as low as such figure at?

---

EDIT

I just found the following answer at Quora:

Of course. A discount factor of 0 will never learn, meanwhile a factor near of 1 will only consider the last learning. A factor equal or greater than 1 will cause the not convergence of the algorithm. Values usually used are [0.2, 0.8]

EDIT: That was the learning factor. The discount factor only affect how you use the reward. For a better explanation:

State-Action-Reward-State-Action - Wikipedia

See influences of variables .

I don't know what is written in the question as it in not visible in Quora, but it seems that the 0.2 to 0.8 figure is used for learning factor, not discount factor. Maybe the author is confused with it...? I'm not sure what the learning factor is, though.

Answer 1

The discount factor is not something you should be optimising. It is typically part of the problem statement.

For practical purposes, you may set it below 1.0 for continuous problems when in fact you care about best long-term reward. Another option to avoid infinities on continuous problems is to re-formulate the problem as optimising average reward. A high discount factor of e.g. 0.99 or 0.999 should produce a similar policy as one based on average reward.

Is the author correct here or does it depend on cases?

The author appears to be either completely wrong, or just poor at explaining themselves on this part.

If it is, then what kind of problems and/or situations should I set the discount factor as low as such figure at?

A low discount factor is for when you care much more about immediate rewards. You set it that low when that is the case. You decide what you care about when you set the learning problem. The value of the discount factor is part of the setup that decides what the optimal policy is. You never set it low "to help with optimising" because changing the value could change the optimal policy.