0
$\begingroup$

It is a trend in deep learning to train models using multi-batches, i.e., to show the model a subset of the entire dataset for each weight update. In some cases, as in continual learning, we see that it is possible to train the network on one datapoint at the time. It is quite rare, on the other hand, to see research papers, or state of the art models, that are trained on a whole dataset simultaneously, i.e., in full-batch training.

For a research project, it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches. An example I have found is the COIN and COIN++ papers, that train on full batch training in order to memorise datapoints.

Question:

Do you know other applications where performing full-batch training is preferable to use mini-batches? Which ones?

$\endgroup$

1 Answer 1

1
$\begingroup$

I think the closest approach to what is described in the papers you linked is Neural Style Transfer. But I see also some (potential) misunderstanding regarding the full batch training, so let me elaborate a bit.

In the papers you linked and the neural style transfer one a model is trained on a single image. Not a single image at the time, literally on a single image. So I wouldn't call it full batch training, since there's no batch at all, and there can't be one. But I guess this is open to interpretation and it depends on how you define a batch in the first place.

it would be useful for me to come up with a list of applications/models where training a neural network in full batch training is preferable with respect to using multi-batches

Aside from these examples (i.e. overfitting or fine tuning on a single training instance) I doubt you'll find papers where people use full batch training. Not only full batch training comes with all sort of computational issues (like keeping in memory all gradients for an entire dataset) but even from a mathematical point of view mini batch and stochastic gradient descent works better than full batch and (classic) gradient descent. I did link some references about this topic in another answer.

$\endgroup$
1
  • 1
    $\begingroup$ Thank you for your answer and reference! It looks interesting. A little comment: while it is true that the COIN paper trains on a single image, it is encoded as W x H different datapoints, one per pixel. This is why I called it full batch. More generally, yes, I'm interested in all of these very edge cases used in practice. $\endgroup$
    – Alfred
    Commented Oct 10, 2022 at 19:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .