I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.

How is that possible? I thought we needed to pad all examples in a batch to model.max_input_size, however, it seems HuggingFace does Dynamic Padding that allows sending batches of different lengths (till the time they are smaller than max_input_size)

enter image description here

Link: https://mccormickml.com/2020/07/29/smart-batching-tutorial/
Link2: https://huggingface.co/learn/nlp-course/en/chapter3/2?fw=pt#dynamic-padding


1 Answer 1


The reason why you want to pad at all is because you want to stuff everything into giant matrix multiplies. GPUs are great at parallelizing these operations. Instead of running e.g., $Wx_i + b$ 32 times, you can run $W X + b$ a single time, where each column of $X$ is a single input.

So, if you have inputs with different input lengths, you just need to add enough padding such that you can fit all your $x_i$s into a single matrix $X$. That is, you only need to pad to the maximum length item in your batch.

The padding tokens are also masked, so that tokens can't attend to them and they don't contribute to the loss. Because of this, adding any additional padding tokens wouldn't really do anything.

  • $\begingroup$ Sorry but my question is how come different batches have different length. In the example pasted above 2 of the batches had length of 13 and 1 had a length of 14. $\endgroup$
    – PS1
    Aug 24, 2023 at 16:03
  • $\begingroup$ Because you only need to pad to the maximum length sequence in your batch. Some batches will have shorter sentences and some batch will have longer sentences. $\endgroup$ Aug 24, 2023 at 17:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .