I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as described in the Atari with Deep RL paper. All is working fine, but I was wondering whether there is any logical way to select the proper batch size for training, or if simply using a grid search is best. At the moment, I’m simply using 32, for its small enough that I can render the gameplay throughout training at a stunning rate of 0.5fps. However, I’m wondering how much of an effect batch size has, and if there is any criteria we could generalize across all Deep Q-learning tasks.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There is no special calculation you can do to determine the optimal batch size for any situation, so you kinda have to do a bit of testing to determine what batch size will work best. But there are some common trends you can take into account to make your testing easier.
How to choose your batch size
According to the paper Accelerated Methods for Deep Reinforcement Learning you get the best performance from DQNs (on average) with a batch size of 512. The problem with this is that is is much slower than the usual batch size of 32 and most of the time the performance improvement doesn't warrant it.
If you are just trying to test out your agents it is generally best to stick with a batch size of 32 or 64 so that you can train the agent quickly yet still get an idea of what it is capable of. But if getting the best performance is your top priority and waiting longer isn't a problem, then you should go for a batch size of 512 (higher can actually lead to worse performance) or something near that.
