How does batch size affect model size?

Question

I'm suffering from a significant brain fart while trying to get my head around how does batch size affect overall model size e.g for CNNs. Does it serve as an additional dimension for all the weight tensors?

Considering:

• VGG16 model
• batch_size of 16
• image size of 224x224x3
• conv_1 being the initial 1x1 convolution with stride 1 and 3:64 channels mapping

The input will be a tensor of [16, 224, 224, 3] shape. Will the output of convolution layer be [16, 224, 224, 64] and therefore - will all the weights have additional 'batch size' dimension and thus - impose a linear increase of model size with respect to the batch size?

Answer 1

The model size (i.e., the number of parameters) is completely independent of the batch size you use during the training.

Whatever the batch size is, your model processes each input within the batch independently. Concretely, batch size affects how many samples your model has to process before it makes an update.

For example, when batch size is 1, it computes the gradients and updates the parameters for each input. When batch size is the whole dataset, it computes the gradients for each input in the dataset independently. It then takes the average of those gradients and uses this average to update the model, so there will be a single update per epoch.

In your case, the model takes an input of size (width=224, height=224, color_channels=3) whatever batch size you use during the training.