Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.
Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.
From what I've read, the meaning of the discount factor (gamma) is to decide "how much we value future rewards."
Correct me if I'm wrong (2):
With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward.
With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from this.
I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself "for the future" when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?
To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a "episode end reward" and a "steps until reward" for each tuple and we in some way put these two new pieces of information into the function.
I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.
Please help :)