1
$\begingroup$

I'm suffering from a significant brain fart while trying to get my head around how does batch size affect overall model size e.g for CNNs. Does it serve as an additional dimension for all the weight tensors?

Considering:

  • VGG16 model
  • batch_size of 16
  • image size of 224x224x3
  • conv_1 being the initial 1x1 convolution with stride 1 and 3:64 channels mapping

The input will be a tensor of [16, 224, 224, 3] shape. Will the output of convolution layer be [16, 224, 224, 64] and therefore - will all the weights have additional 'batch size' dimension and thus - impose a linear increase of model size with respect to the batch size?

$\endgroup$
  • $\begingroup$ You probably mean not "model size" winch is not affected but amount of working memory, which is model size + layers output + temporary variables. It depend on framework/implementation $\endgroup$ – mirror2image Feb 13 at 6:27
1
$\begingroup$

The model size (i.e., the number of parameters) is completely independent of the batch size you use during the training.

Whatever the batch size is, your model processes each input within the batch independently. Concretely, batch size affects how many samples your model has to process before it makes an update.

For example, when batch size is 1, it computes the gradients and updates the parameters for each input. When batch size is the whole dataset, it computes the gradients for each input in the dataset independently. It then takes the average of those gradients and uses this average to update the model, so there will be a single update per epoch.

In your case, the model takes an input of size (width=224, height=224, color_channels=3) whatever batch size you use during the training.

| improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ That makes perfect sense, but how is it processed in terms of actual computations on GPUs? Processing samples in batch iteratively seems like very inefficient way of completing the task. $\endgroup$ – pSoLT Feb 12 at 13:46
  • $\begingroup$ So since batch gradients are independent, GPU processes them in parallel. $\endgroup$ – SpiderRico Feb 13 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.