As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is set, and to allow batching, all sequences smaller are padded.

However, after a Transformer is trained, and we want to run it on a single sequence at inference time, the computational costs are far less than training. Thus, it seems reasonable that I would want to run the transformer on a larger input sequence length during inference time. From a technical perspective, this should be feasible.

I keep reading online that a Transformer cannot be run on a sequence size larger than the one seen during training. Why is this? Is it because the network weights will be unfamiliar with sequences of this length? Or is it more fundamental?


To some extent, this is true; The piecewise feedforward layers can be added or subtracted to fit the sequence length. The matrix operations can similarly be scaled to fit sequence length.

However, the computational complexity comes from the matrix operations in the attention layer. Those are not trained; There are no trained parameters in the attention mechanism (see figure 2 and equation (1) in Vaswani et al). So, those have to be computed during inference as well.

Another challenge would be the output layer. That layer is a regular feedforward layer and thus has a fixed input size; That is, you cannot add new parameters during inference.

Of course, there is a caveat to this; There are now transformers now that allow recurrence, such as Transformer-XL and Memformer. These do, in a way, allow longer input sequences than "max sequence length".

  • $\begingroup$ I think the part about the output layer is incorrect. Usually the output layer is repeated for thesequence length, so that should not be an issue. Also the point about the matrix operations does not answer why transfoermers CANNOT be used for sequences larger than what they were trained on. $\endgroup$ Nov 29 '21 at 22:16
  • $\begingroup$ @chessprogrammer It's not strictly necessary to repeat it for the sequence length; It helps with training, but the output is in principle a single token (especially if you want to avoid teacher forcing, although that's of course still ongoing research). As Vaswani say in the first paragraph of section 3 in their paper: "Given z, the decoder then generates an output sequence (y1, ..., ym) of symbols one element at a time." $\endgroup$
    – Avatrin
    Nov 29 '21 at 22:31
  • $\begingroup$ @chessprogrammer I only mentioned the matrix complexity point because I interpreted your question as being premised on the quadratic complexity being the reason for the limitation. $\endgroup$
    – Avatrin
    Nov 29 '21 at 23:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.