How does the discount factor (gamma) and memory replay work in Deep Q-learning?

Question

Maybe in over my head about this but I'm having a hard time understanding the discount factor in Deep Q-learning.

Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.

From what I've read, the meaning of the discount factor (gamma) is to decide "how much we value future rewards."

Correct me if I'm wrong (2):
With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward.
With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from this.

I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself "for the future" when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?

To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a "episode end reward" and a "steps until reward" for each tuple and we in some way put these two new pieces of information into the function.

I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.

Please help :)

Answer 1

Let's try to understand what actually happens when learning. You start with random values for Q(s,a), the function that estimates the reward of an action given a state s, and you then get an immediate reward and then your loss is difference between the given reward + gamma *future estimated reward, i.e. $loss = Q(s,a) - (reward + gamma * Q(s', max_a))$.

Now if gamma=0, it means the loss would not be dependent on what happens afterwards - since the loss would simply be $Q(s,a) = reward$. So eventually Q(s,a) would simply converge to be the immediate reward of action a in state s. This means the agent would not end up learning the optimal strategy, since it would choose the action that maximizes the immediate reward without caring what happens afterwards, even if the step after it will die.

In order to get some intuition on how in practice it actually helps the agent look at the future, try to imagine how this would work out in practice. In the beginning rewards would be random, and at some point it will get to a state before the final episode reward would be very positive if the agent won. Then at some point it would get to the 2 before last state and might choose the action that gets to the positive one before last state - in which case since gamma > 0 it would optimize Q(s,a) to something positive. After millions of s,a pairs you can imagine how this will create a good agent.

Regarding your other suggestion - I am not an expert but I imagine there might be ways to take into account the final episode reward in a different manner - but the main nice thing about DQN is that it is simple and local - and even though you just update based on samples from local actions, with time the network will (hopefully) converge to be optimal.