I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.
I have a very simple environment, basically a 18x18 matrix, where 3 objects live. It is basically a penalty shootout, one player, one ball and one keeper. The player should learn to score goals. The player can move forward, to the left, to the right, and 45° left and 45° right. The keeper moves left and right in front of his goal.
I already used a CNN approach where I fed the 18x18 matrix as image, onto three layers with 64 units each and let the agent learn. Then I used a network with one input layer and 38 features, three hidden layers with 64 units each, and finally I used 2 hidden layers with 256 units each. ReLU and Adam.
All approaches worked. Now, I want to find out which approach works "best". But I don't know in which direction to go. All of the training sessions so far took considerable long time. The last approach e.g. takes a few days till the agent figures out not to move of the environment, that he needs to hit the ball, and in which direction he need to shoot the ball.
During the training sessions, I need to adjust the reward function in order to improve the agent. I start with a learning rate of 0.01 and then I reduce it to 0.001 after 300.000 episodes, each limited to 100 steps.
I read about grid search, but I expect this needs enormous amount of time, and I don't have access to lots of processing power, only a simple laptop style GPU.
What is the strategy to get to a better network?