2
$\begingroup$

From what I understand, Transformer Encoders and Decoders use a fixed number of tokens as input, e.g., 512 tokens. In NLP for instance, different text sentences have a different number of tokens, and the way to deal with that is to truncate the longer sentences and pad the shorter ones. As an additional input, a padding mask must be supplied to the Transformer so that its attention is only focused on the relevant tokens.

My question is: Is there something in the architecture that forces the transformer to have a fixed number of tokens as input? (and not adopt dynamically to the actual input length like RNNs for instance?)

For comparison, I think of fully-convolutional networks or RNNs with variable input lengths. They are agnostic to the actual input dimension because they perform pointwise operations on the different patches. When applying an RNN model to an n-tokens sentence, you compute the same block n times, and when computing it on a k-tokens sentence you will apply it k times. So this architecture does not require padding or truncating (at least not in theory, I do not refer here to implementation considerations). In transformers: embedding the tokens, computing attention, and feed-forward can be performed on different lengths of sequences since the weights are applied per token, right? So why do we still truncate and pad to a fixed size? Or perhaps it is feasible but not implemented in practice for other reasons?

I must be missing something...

I'll ask it differently to make my question more clear: Say I have an already-trained transformer model, trained on 512 fixed-sized inputs (truncated and padded). At inference time, if I would like to process a single, shorter sentence. Do I have to pad it or not?

Thanks

$\endgroup$
2
  • 2
    $\begingroup$ It's important to remember that all machine learning and deep learning is at the core matrix algebra and multiplications. So it makes lot of sense to have fixed dimensions of matrices. $\endgroup$ Oct 25, 2022 at 14:05
  • $\begingroup$ You mean that the reason for forcing a fixed sentence length is for computation performance? $\endgroup$
    – A. Maman
    Oct 26, 2022 at 6:48

2 Answers 2

3
$\begingroup$

Edits to reflect edits in question: If you train your transformer on length = n then yes, you need to pad inputs to length = n. This is not a requirement in the mathematical architecture, it's a requirement in the implementation.


There seem to be two separate ideas in your question:

  1. Why do transformers have a fixed input length?
  2. Why do transformers have input length limitations?

I am not sure which one you are asking, so I will answer both.


1) Saying transformers have a fixed input length is misleading.

Transformers accept variable length inputs just like RNNs. You can think of padding/truncating as an extra embedding step if you want.

We don't need padding in RNNs because they process inputs (sentences) one element (token/word) at a time.

Transformers process inputs all at once. If you are passing in several sentences, you have to do something to regularize sentence length, hence padding and truncating.


2) Transformers often limit input length to 512 or 1024 because of performance issues.

If you are wondering why we don't let our transformers accept inputs as long as possible, the answer is that there are computational and performance limitations. The algorithm complexity is quadratic.

This is where the max_length parameter of a transformer comes in. If your input has length 1,000, the transformer will throw an error because it can only handle inputs up to 512 in length.

$\endgroup$
15
  • $\begingroup$ I’m not sure I understood what you mean. If it’s the tokenization algorithm that you’re talking about, so that is not what I’m asking. I doubt the limit on the sequence length of the model (ie the number of tokens, regardless of the embedding dimension or the vocabulary size. $\endgroup$
    – A. Maman
    Oct 26, 2022 at 6:51
  • $\begingroup$ @A.Maman I just updated my answer to be more detailed. I hope it helps, but if not, you might want to provide an example in your question so we know exactly what you are asking :) $\endgroup$
    – waxalas
    Oct 26, 2022 at 15:30
  • $\begingroup$ @A.Maman I just realized maybe the answer to your question is that RNN are recurrent, processing one word at a time along with the previous state. They do "pointwise operation" like you said. Transformers try to solve this extremely expensive method by processing whole sentences at a time, without the need for recursion. That's why you need to choose an input length from the start, because you are feeding the complete sentence at once. $\endgroup$
    – waxalas
    Oct 26, 2022 at 16:11
  • $\begingroup$ Hi! thanks for the answer! I understand that max_length is not constant in a sense of being numerically identical among all transformers and that it is actually a parameter of the architecture. That's why you're right that it is more accurate to say that it is "fixed" and not "constant". I edited my question to try making it more clear but what I basically ask is: if we disregard training matters, do transformers know to process different lengths of inputs or not? because how I understand the architecture, they do. And if so, whey do I have to pad a short sentence to process it?thanks! $\endgroup$
    – A. Maman
    Oct 27, 2022 at 10:58
  • $\begingroup$ regarding the matrix multiplication explanation: I understand that weight matrices have fixed dimensions. But all those matrices in transformers work pointwise on each token (embedding matrix, query, key, value projections, and feed-forward is performed pointwise as well) , and the token embedding dimension remains the same regardless of the sequence length. So that should not be a problem for processing variable lengths. $\endgroup$
    – A. Maman
    Oct 27, 2022 at 11:05
1
$\begingroup$

To add something to pip.pip answer (thumbs up cause is totally on point), consider that transformers can't be fully convolutional, since as the name suggest, a fully convolutional model perform only convolutions, while transformers include dense layers, which expect a fixed input dimension.

Despite being possible to overcome the limit of fixed dimensionality imposed by the dense layers, for example by using pyramid pooling, that would only add complexity to the training regime and there's no guarantee that the performance will increase.

lastly, from a linguistic perspective, 512 and 1024 tokens are already quite a lot to learn most long dependencies (which let's recall was the main issue why transformers were introduced in place of RNNs). So the game of making transformers input size independent is not really worth the effort.

$\endgroup$
1
  • $\begingroup$ Hi @Edoardo Guerriero Thanks! I didn't mean to ask whether transformers can be fully-convolutional or not. I meant to compare fully-convolutional networks' ability to process different sizes of input resolutions to transformers' in-practice training method of processing fixed-size sequence lengths. That is: can the same instance of a transformer process n-sequence input and k-sequence input without padding. And I mainly ask whether this is theoretically possible by the architecture and not from memory/compute resources or implementation perspective. Thanks again! $\endgroup$
    – A. Maman
    Oct 27, 2022 at 11:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .