How do I design the network for Deep Q-Network?

Question

I am playing with deep q-learning and I am thinking about what the proper architecture of a network used for deep q-learning is.

I have a very simple environment, basically a 18x18 matrix, where 3 objects live. It is basically a penalty shootout, one player, one ball and one keeper. The player should learn to score goals. The player can move forward, to the left, to the right, and 45° left and 45° right. The keeper moves left and right in front of his goal.

I already used a CNN approach where I fed the 18x18 matrix as image, onto three layers with 64 units each and let the agent learn. Then I used a network with one input layer and 38 features, three hidden layers with 64 units each, and finally I used 2 hidden layers with 256 units each. ReLU and Adam.

All approaches worked. Now, I want to find out which approach works "best". But I don't know in which direction to go. All of the training sessions so far took considerable long time. The last approach e.g. takes a few days till the agent figures out not to move of the environment, that he needs to hit the ball, and in which direction he need to shoot the ball.

During the training sessions, I need to adjust the reward function in order to improve the agent. I start with a learning rate of 0.01 and then I reduce it to 0.001 after 300.000 episodes, each limited to 100 steps.

I read about grid search, but I expect this needs enormous amount of time, and I don't have access to lots of processing power, only a simple laptop style GPU.

What is the strategy to get to a better network?

Answer 1

What is the strategy to get to a better network?

There are a few different strategies that you can use to search for good hyperparameters in reinforcement learning RL, but you should be aware that this is a hard problem:

• Even with supervised learning on fixed datasets, efficient hyperparameter searches are not a solved problem.

• RL adds more hyperparameters for exploration vs exploitation, and management of the training data (e.g. in DQN, the replay memory size, how often to update the target network). These hyperparameters also need to be tuned and will interact with NN-specific hyperparameters.

• RL requires data collection as part of the learning process, and this can be more expensive than training any neural network internal within the agent. As well as simulating or sensing the environment and on-the-fly feature engineering, this usually requires running the network forward in order to enact the current policy, separate from prediction/loss calculations.

• RL control involves learning then forgetting "interim" functions in a sequence of gradual improvement. Compared to learning an optimal policy directly in a supervised manner, this can take many times more effort.

• In RL, learning curves can be highly variable, making quick decisions based on early performance unreliable.

It is not much a surprise given all this, if a bit disappointing for solo researchers, that cutting edge RL on complex problems is mainly handled by large organisations with deep pockets to cover the compute costs.

As a hobbyist, I don't know or have access to large scale solutions, where some new strategies could open up due to the amount of compute available. Assuming you are in a similar situation, perhaps some of the following could help:

• First and foremost, decide what your goals are for with the current experiment. In different scenarios you will want to focus on certain kinds of change:

  • Is it to make the best agent possible for a target environment?
  • Is it to learn a specific RL technique?
  • Is it to test the limits of a specific RL method or agent?
  • Is it to noodle around with the agent and environment to gain general experience about how the combination behaves when you vary things?

• For "best agent" scenarios, learning the problem domain and feature engineering based on that knowledge could help more than hyperparameter tweaking.

• If you are not in a "best agent" scenario, consider whether you have learned enough from the current experiments and move on to some other environment.

• If you and your computer have nothing better to do, and want to explore possible improvements, arrange for experiments to run when you are not using the computer for other projects, e.g. overnight, and keep an editable "TODO" list of next things to try. This could be simple grid searches (if training is relatively quick) or ideas for hyperparameters inspired from theory, effectively random search, third-party suggestions or simply to see what happens.

  • Automate as much of this as possible, including putting checkpoint files in dedicated per-experiment folders, using some kind of versioning system to label the experiments.

• With some experience (maybe from just "noodling around" on this or other projects) you may spot learning patterns/behaviours that imply changing specific hyperparameters would be useful. This knowledge does not always transfer when systems (or other hyperparams) are very different, but often enough it does.

• You can pre-test some NN architectures for capacity to learn a policy or value function by making a training dataset based on the policy or returns for a reasonably good agent - either an expert or a previous RL experiment - then choosing by loss or accuracy or some other metric. This will mainly help you rule out poor architectures without running a full RL training. It probably won't help you to select the best out of the good ones, because once a NN has capacity to learn your test data within reasonable accuracy, differences that you cannot test this way will become more important.

• One good rule of thumb is to avoid large/complex NNs, and search for the simplest NN that learns your target function well. This will speed up training times, although it does mean you will rarely explore what happens with larger more sophisticated networks.

• It is often worth working on code optimisation for speed, if your CPU time is a constraint, then extra developer time optimising the environment and the agent will pay back well, allowing you to run more experiments. You can look into Python toolkits like numba (which compiles a limited set of Python to C functions) to speed up custom environments.

  • Related to this, disabling visual output from the environment during training can give you a large speed improvement. You can switch it back on when reviewing agent behaviour.

There are more advanced hyperparameter searches than grid search that may be promising (including use of Bayesian statistics, or genetic algorithms, or even more RL to set hyperparameters), but when your compute budget is low you may find educated guessing and some random search works just as well or better.