I am trying to run a code that has a batch size around 28. I can run the program on my GPU with this batch size.
But, when I modify the code for my requirements and try to run, it is showing an run-time error due to insufficient memory in GPU.
I checked for possible batch-size that I can run and it is just 2-5.
I am not sure whether there is any issue if I run with such small batch sizes? I mean, will there be any performance issues keeping aside the time it takes?
Batch size affects how many training updates (steps) will happen during each epoch.
When the batch size is small, this means that the model sees fewer data in each weights update. Thus, your question really depends on the data you have, along with the corresponding task (classification / RL etc.)
If your data is highly imbalanced, then I would not suggest a small batch size, since the probability of seeing a positive instance would be far smaller (assuming you take uniform batches).
For an RL task, imagine using a replay buffer of past experiences and your agent had very few good action selections during the only exploration process. Then a small batch size would make the agent training very difficult, since most of the time samples with not good action selections would be seen. As a result, the agent may drift from good policies.
For a classification task, what I always do, is to make stratified batches. That is each batch has the same label percent as the whole dataset. And most of the time it works for better than uniform batches even for smaller batch sizes. For RL, I would recommend higher batch sizes or similar clever ways of sampling.
