I am trying to understand how to train a neural network to win a Pong game using reinforcement learning, by following the blog post Spinning up a Pong AI with deep reinforcement learning.
The environment is provided by Gym AI. It gives the AI a reward of 1 if the opponent misses the ball, and a reward of -1 if it misses the ball.
I am confused about how reward discounting works in this context. This is the function that the blog post used:
def discount_rewards(r, gamma): """ take 1D float array of rewards and compute discounted reward """ r = np.array(r) discounted_r = np.zeros_like(r) running_add = 0 # we go from last reward to first one so we don't have to do exponentiations for t in reversed(range(0, r.size)): if r[t] != 0: running_add = 0 # if the game ended (in Pong), reset the reward sum running_add = running_add * gamma + r[t] # the point here is to use Horner's method to compute those rewards efficiently discounted_r[t] = running_add discounted_r -= np.mean(discounted_r) #normalizing the result discounted_r /= np.std(discounted_r) #idem return discounted_r
Basically, the list of rewards is mostly filled with zeros, because usually nothing happens. When something happens, e.g. the reward is 1, this is not only due to the action taken in that step. Therefore, we need to smoothen the list of rewards so that some of that reward also belongs to previous actions. So far so good. However, it seems to me that if the opponent misses the ball and the reward is 1, then this will be smeared such that it will emphasize the actions taken right before the opponent missed the ball. This seems wrong to me, the actions taken by the AI right before the opponent missed the ball are irrelevant. They don't affect the ball in any way.
I only think reward discounting makes sense when you have lost a point, then the actions just preceding the loss are surely very important and should be emphasized. However, the function takes into account both wins and losses.
How should reward discounting be understood in the context of this Pong game?