I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-pasted the code exactly as it is but added some lines to get the graph on figure 18-10 presenting the sum of total rewards gained during each episode.
So everything is the same as the book except the training part where I added the lines related to rewards lists and the plotting:
all_rewards =  for episode in range(600): obs = env.reset() episode_rewards =  for step in range(200): epsilon = max(1 - episode / 500, 0.01) obs, reward, done, info = play_one_step(env, obs, epsilon) episode_rewards.append(reward) if done: break all_rewards.append(episode_rewards) if episode > 50: training_step(batch_size) sum_rewards =  for i in range(len(all_rewards)): sum_rewards.append(sum(all_rewards[i])) import matplotlib.pyplot as plt episodes = range(1,601) plt.plot(episodes, sum_rewards)
At my surprise I didn't get the same graph as the one the author presents in its book, so I reran the code again and got a totally different graph from what I had for the first time. Please find below two graphs that I obtained. I'm plotting the total sum obtained during each episode with respect to the episode.
I'd like to ask you if there is some intrinsic to the algorithm that makes it so random and in that case I'd like some references (if there are any) that prove that or I'm just doing something wrong. Thank you.