# How does the discount factor (gamma) and memory replay work in Deep Q-learning?

Correct me if I'm wrong (1):
To train a Deep Q-learning network, every N:th step of action taken, a batch of random samples from replay memory are fed into the model's training function. These samples are totally unrelated to each other, they are unordered and we can't tell if a particular sample is taken from an episode which ended with a positive or negative reward. This is all there is to the training part of a Deep Q-learning network.

From what I've read, the meaning of the discount factor (gamma) is to decide "how much we value future rewards."

Correct me if I'm wrong (2):
With gamma = 0: The model only cares about the reward from when the state goes from A to B given the action C, and the training is encouraging this behaviour of the model with the reward.
With gamma = 0.9: The model cares about future rewards and acts based on what might come 5 (or what number we might want) steps from this.

I know I have gotten something wrong somewhere (or on multiple places :D): How can this small value, gamma, make the model train itself "for the future" when nobody (not even human) kan tell what the reward 5 steps from a particular step will be?

To me it would have made sense if we had not only 32 random tuples of (state, action, reward, state') but also a "episode end reward" and a "steps until reward" for each tuple and we in some way put these two new pieces of information into the function.

I'd really like to understand what the gamma does and how. I'm far from a maths person but from the code I've seen in a few tutorials and examples, I can't even begin to understand how this 0.9 or 0.99 can enable the model to train for future reward.

• You may need to show some of the equations you are looking at. You refer multiple times to a variable called "sigma", but I am not familiar with a use for a variable labelled with $\sigma$ or $\Sigma$ in the usual equations of reinforcement learning or deep Q learning. Nov 19, 2022 at 9:17
• Hello again @NeilSlater, I'm sorry, as I wrote I suddenly started to name the factor "sigma" when it should have been "gamma". I've now corrected the original post. Again, I'm sorry for the mistake, my only excuse being that I was to tired after trying to fix my neural net all day. Nov 19, 2022 at 17:18

Let's try to understand what actually happens when learning. You start with random values for Q(s,a), the function that estimates the reward of an action given a state s, and you then get an immediate reward and then your loss is difference between the given reward + gamma *future estimated reward, i.e. $$loss = Q(s,a) - (reward + gamma * Q(s', max_a))$$.
Now if gamma=0, it means the loss would not be dependent on what happens afterwards - since the loss would simply be $$Q(s,a) = reward$$. So eventually Q(s,a) would simply converge to be the immediate reward of action a in state s. This means the agent would not end up learning the optimal strategy, since it would choose the action that maximizes the immediate reward without caring what happens afterwards, even if the step after it will die.