4
$\begingroup$

I am training an RL agent using Deep-Q learning with experience replay. At each frame, I am currently sampling 32 random transitions from a queue which stores a maximum of 20000 and training as described in the Atari with Deep RL paper. All is working fine, but I was wondering whether there is any logical way to select the proper batch size for training, or if simply using a grid search is best. At the moment, I’m simply using 32, for its small enough that I can render the gameplay throughout training at a stunning rate of 0.5fps. However, I’m wondering how much of an effect batch size has, and if there is any criteria we could generalize across all Deep Q-learning tasks.

$\endgroup$

2 Answers 2

1
$\begingroup$

There is no special calculation you can do to determine the optimal batch size for any situation, so you kinda have to do a bit of testing to determine what batch size will work best. But there are some common trends you can take into account to make your testing easier.


How to choose your batch size

According to the paper Accelerated Methods for Deep Reinforcement Learning you get the best performance from DQNs (on average) with a batch size of 512. The problem with this is that is is much slower than the usual batch size of 32 and most of the time the performance improvement doesn't warrant it.

Performance vs. Batch Size

If you are just trying to test out your agents it is generally best to stick with a batch size of 32 or 64 so that you can train the agent quickly yet still get an idea of what it is capable of. But if getting the best performance is your top priority and waiting longer isn't a problem, then you should go for a batch size of 512 (higher can actually lead to worse performance) or something near that.

$\endgroup$
0
$\begingroup$

In my experience, exploration is always the challenge with RL, and I think about batch size a way of increasing exploration.

What happens during training of an RL agent, is the policies entropy often reduces as the agent receives rewards, leading to reduced exploration. If the rewards in the environment are sparse, the agent can sometimes only discover the first reward, and it can take time to discover other rewards.. as it just beelines to the reward it knows about, and doesn't explore enough.

Increasing batch size, (assuming your not using a single environment, but many parallel environments) has the effect of ensuring that the agent can get multiple reward signals from more diverse state transitions... allowing a more accurate value function and therefore a more complex policy that explores more. The challenge is that finding the correct learning rate can be tricky, as well as the values of other hyperparameters.

Overall, I would say for something like Atari, which is a very high dimensional state space, aim for the largest batch size you can get the agent to learn at.

For reference, I'm running a batch size of 4096 a gridworld maze exploration task (using PPO), and I'm getting very good results at that size. I would go larger but I don't have the memory to do so, however I had to run at least 40 hyperparameter sets to find the settings that worked at that batch size.

Finally, I would point out that advantage based algos are a lot more stable, as the values are implicitly normalized, which probably is why I'm able to reach such a big batch size with so few hyperparameter adjustments. You may find DeepQ to be a lot harder to stabilize.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .