0
$\begingroup$

Background: In C51 DQNs you must specify a v-min/max to be used during training. The way this is generally done is you take the max score possible for the game and set that to v-max, then v-min is just negative v-max. For a game like Pong deciding the v-min/max is simple because the max score possible is 20, therefore, v_min=-20 and v_max=20.

Question: In a game like Space Invaders, there is no max score, so how would I calculate the v-min/max for a C51 DQN?

$\endgroup$
1
$\begingroup$

If you're using a discount factor less than 1, you should be able to compute a maximum return (likewise, a minimum return) based on the max (min) reward you can earn at each timestep. However, this issue you bring up is usually cited as a difficulty with C51. I think people tend to simply use fixed values for the min/max return (or just make rough estimates). If you want to avoid this, I recommend looking into the QR-DQN algorithm which circumvents this issue altogether and is more theoretically sound.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.