I am trying to solve the game of Hanabi (paper describing game) with actor-critic algorithm. I took code for the environment from the Deepmind's repository and implemented a2c algorithm myself. From the tests on simple gym environments (lunar lander, cartpole) my implementation takes about the same time to converge as A2C from OpenAI/baselines. So, I suppose that algorithm is implemented correctly and problem is somewhere else.
In the paper, actor critic algorithm used is IMPALA, but I am currently using classic A2C, because I thought that difference should not be that crucial. Also, I do not use population based training yet, but in Hanabi paper there is plot showing that even with out PBT their algorithm shows decent results.
Basically, problem is that I can not even solve so called "Hanabi-Very-Small" which is a simplification (state space of dim.100 instead of 600). It hits 3 points out of 5 after some training and then learning curve saturates and on the full environment learning stops at ~5.8 pts/25
I have been playing with hyper parameters, such as learning rate, gradient clipping and weight of entropy term in the loss function but it did not help. Architecture is fc256-lstm256-lstm256, same as in Hanabi paper.
Since I do not have much experience in RL, I am confused with this behaviour and do not know what is the reason, so I am asking for hints on where the problem could be.
Firstly, can it be simple because IMPALA is better algorithm in general? I expected A2C to work worse, but 5.8 vs 24 seems to be to much of a difference.
Another question is how to search for optimal hyper-parameters effectively? Learning, even for very small version of the game, takes several hours until it saturates and doing greed search would take to long? Are there any common heuristics that could help? Also, is the choice of hyper-parameters that important? I mean can it really change behaviour from "largely undefits" to "nearly perfect" ?
Thanks in advance!