I am using the default implementations of REINFORCE, DQN and c51 available from the tf.agents repo (links). As you can see, DQN manages to improve performance while REINFORCE seems to suffer from catastrophic forgetting. OTOH, c51 is not able to learn much and performs like a random policy throughout.

The environment looks like this -

  • action = [66, 1]
  • states = [20, 1]
  • max possible state value = 20
  • steps per episode = 20
  • Hidden Layer dimension = (128, 128)
  • learning rate = 0.001 (constant throughout)
  • Epsilon (exploration factor) = 0.2 with decay of 0.05 every 4000 episodes
  • Discount factor = 0.9
  • relay memory size = 10,000

Every episode runs for 20 steps and the rewards are collected for every step.

actual episode value this x-axis value multiplied by 50 actual episode value is the plot is the x-axis value multiplied by 50

What could be the possible reasons for such a performance of c51 and DQN? And based on the state space, are my hyperparameters correct or some of them need more tuning? I will increase the replay memory size but other than that to check for catastrophic forgetting, I am not sure how to diagnose other issues.



You must log in to answer this question.