I have a scenario where, in an ideal situation, the greedy approach is the best, but when non-idealities are introduced which can be learned, DQN starts doing better. So, after checking what DQN achieved, I tried C51 using the standard implementation from tf.agents (link). A very nice description is given here. But, as shown in the image, C51 does extremely bad.

enter image description here

As you can see, C51 stays at the same level throughout. When learning, the loss right from the first iteration is around 10e-3 and goes on to 10e-5, which definitely impacts the change in the weights. But I am not sure how this can be solved.

The scenario is

  • 1 episode consists of 10 steps and the episode only ends after the 10th step, the episode never ends earlier.

  • states at each step are integer values and can take values between 0 and 1. In the image, states are of shape 20*1.

  • actions have the shape 20*1

  • learning rate = 10e-3

  • exploration factor $\epsilon$ starts out at 0.2 and decays up to 0.01

C51 has 3 additional parameters, which help it to learn the distribution of q-values

num_atoms = 51 # u/param {type:"integer"} 
min_q_value = -20 # u/param {type:"integer"} 
max_q_value = 20 # u/param {type:"integer"}

num_atoms is the number of support that the learned distribution will have, and min_q_value and max_q_value are the endpoints of the q-value distribution. I set them as 51 (the first paper and other implementations keep it as 51 and hence the name 51), and the min and max are set as the min and max possible rewards.

So, if anyone could help me with fine-tuning the parameters for C51 to work, I would be very grateful.

  • 1
    $\begingroup$ I guess you haven't tried yet to use some kind of hyper-parameter optimization technique (such as random search or a grid search), which will probably be costly, but, without domain knowledge, that may be one solution to your problem, assuming that you do not have bugs in your code and C51 isn't supposed to work (well) on this problem with the given hyper-parameters. I am currently not familiar with C51, so I can't help you further. $\endgroup$ – nbro Nov 21 '20 at 22:13
  • $\begingroup$ @nbro I see, I used the standard implementation from the website linked and only changed the num_atoms, min_1_value, max_q_value along with the replay memory size. So yes I haven't tweaked a lot of things around it. I will see if I could use any hyper-parameter optimization technique, should be a good learning experience. $\endgroup$ – user3656142 Nov 21 '20 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.