Is it possible/advised to use Mini-Batch like accumulation with Adam optimization?
How would that works?
Do I accumulate the loss function for each sample in the batch and then run Adam, or should I divide the Loss by Number of samples in batch before running Adam?
Adam works best with mid-sized mini batches.
Too small batches can generate too much sampling noise, making Adam less stable than basic stochastic gradient descent.
Too large batches remove the computational advantages of using mini batches in the first place and slow learning.
Important: For gradient estimates in neural networks you cannot simply accumulate or average loss values. You must calculate the full gradient of the loss for each example (using back propagation). You can then aggregate and take the mean of these high dimension gradients.
Accumulating or taking mean gradients are almost the same thing in Adam, provided the batch size remains constant. That's because Adam divides all gradient components by running averages.
However, using the mean gradient is closer conceptually to what is going on: The mean gradient of a mini batch is an approximate estimate of the full batch mean gradient. Adam then processes a sequence of these estimates into update steps that roughly account for second-order gradients.
You can use Adam (and other gradient-based optimisers) separate to sampled data. E.g. when you want to find a minimum of a function that you know, instead of a mini batch, calculate the true gradient at a current position and use that gradient as the input to Adam. If you have not done that before, it's worth experimenting with for multiple gradient descent accelerators, because it helps understand how they are separate to sampling and how they work
