0
$\begingroup$

I noticed something rather intriguing while testing the Deep Q-Network implementation from Aurélion Géron's book Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition; I copy-pasted the code exactly as it is but added some lines to get the graph on figure 18-10 presenting the sum of total rewards gained during each episode.

So everything is the same as the book except the training part where I added the lines related to rewards lists and the plotting:

all_rewards = []
for episode in range(600):
  obs = env.reset()
  episode_rewards = []
  for step in range(200):
    epsilon = max(1 - episode / 500, 0.01)
    obs, reward, done, info = play_one_step(env, obs, epsilon)
    episode_rewards.append(reward)
    if done:
      break
  all_rewards.append(episode_rewards)
  if episode > 50:
    training_step(batch_size)

sum_rewards = []
for i in range(len(all_rewards)):
  sum_rewards.append(sum(all_rewards[i]))

import matplotlib.pyplot as plt
episodes = range(1,601)
plt.plot(episodes, sum_rewards)

At my surprise I didn't get the same graph as the one the author presents in its book, so I reran the code again and got a totally different graph from what I had for the first time. Please find below two graphs that I obtained. I'm plotting the total sum obtained during each episode with respect to the episode.

First graph

Second graph

I'd like to ask you if there is some intrinsic to the algorithm that makes it so random and in that case I'd like some references (if there are any) that prove that or I'm just doing something wrong. Thank you.

$\endgroup$
7
  • $\begingroup$ I’d wager you’re using an $\epsilon$-greedy policy which is random, you’d need to set the same seed as the book to get the same results. $\endgroup$ Jul 13 at 21:46
  • $\begingroup$ @DavidIreland that's true. There is an $\varepsilon$-greedy policy in play. And I saw that the notebook provided on the Github of the author setting the seed at 42, but I find it rather intriguing that the results depend so much on that. Or is it considered as a hyperparameter? $\endgroup$
    – Daviiid
    Jul 13 at 22:05
  • $\begingroup$ it is a hyper-parameter, but the results should not wildly depend on setting a seed. It is likely the author tuned a seed to have good results from the book as DRL can be quite sensitive to hyper parameter selection. Have you tried training for a longer amount of time? you might need to play around with some of the other hyper parameters to get a similar performance to the author on an arbitrary seed. $\endgroup$ Jul 13 at 23:56
  • $\begingroup$ What is the environment? Is it perhaps something that fails/ends early with a mistake? In which case, I don't think the seed is acting as a meaningful hyperparameter here, nor is 42 "tuned" - you are likely to be seeing the learning acting as intended. What is probably wrong is your expectations on your current plots - if you want to see true progress on the learning challenge you have set then you need to measure something else $\endgroup$ Jul 14 at 8:09
  • $\begingroup$ @DavidIreland I have tried a training a longer amount of time but it's of no use. It's still a bit random. For the environment it's CartPole-v0 from gym library. Each episodes is played 200 steps if it's not ended because of the pole falling or else. I agree that we have to examine other metrics but I find it weird how sometimes in 600 episodes the algorithms reaches that 200 reward (eventhough it may fall after) and sometimes even in 2000 episodes it is not the case. I wonder if there are any "proofs" of why it's intrinsically so random. $\endgroup$
    – Daviiid
    Jul 14 at 12:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.