The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and GRUs.

However, having gone through the paper, as well as several online explanations, I still have trouble wrapping my head around how they work. How can a non-recurrent structure be able to deal with inputs of arbitrary length?

  • 4
    $\begingroup$ Transformer cannot handle arbitrary length. You must have misunderstood something. For example, BERT is a transformer architecture (encoder only), and it has a maximum length of 512 tokens. A very useful blog for understanding transformers is The Illustrated Transformer $\endgroup$
    – Astariul
    Commented Aug 10, 2020 at 5:41

2 Answers 2


Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.

However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-attention, therefore, causes no influence. Let's say we use a transformer model with 512 limit of sequence length, then we pass a input sequence of 103 tokens. We padded it to 512 tokens. In the attention layer, positions from 104 to 512 are all masked, that is, they are not attending or being attended.

  • $\begingroup$ Thanks for the answer. Could you clarify one more point: What do you do when the input sequence is longer than 512/1024? Is a transformer simply inappropriate in that case? $\endgroup$ Commented Aug 11, 2020 at 13:35
  • 9
    $\begingroup$ Yes, it's the limitation of BERT. Actually, there are tons of researches trying to solve this issue such as SpanBERT, Longformer, Reformer, Sparse transformer, etc. We could categorize them into two categories roughly. One is embracing the length limitation and divide the sequence into sections, while the others assume the attention could be sparsified. $\endgroup$
    – tsu
    Commented Aug 12, 2020 at 7:12
  • $\begingroup$ @tsu is that limitation still relevant to GPT 4? $\endgroup$
    – Weier
    Commented Feb 28 at 14:49
  • $\begingroup$ @Weier After ~4 years, the limitation has been pushed up to 128K by OpenAI and 100M claimed by Google. IIUC, the main architecture of LLMs is still multi-head attention based transformer although a variety of researches on sparse and efficient transformers are proposed. I would say the main improvement on pushing the sequecne limitation might be attributed to quatization and the optimization on GPUs. $\endgroup$
    – tsu
    Commented Feb 29 at 16:11

The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum sequence length supported by the transformer you have built.

We have found it useful to wrap our transformer in a class that allows us to programmatically use a sliding window across inputs that are longer than the supported transformer input length. If the input is less than or equal to the supported length, it is simply processed. If it is longer, we iteratively slide across the data, passing each window into the transformer and then aggregate the outputs.

When we take this approach, we do not typically slide the window one element (word embedding) at a time, but instead use a longer stride, usually two to five embeddings at a time. We have been intending to do some research into evaluating whether using multiple strides improves overall performance, but have not yet done so because of the prohibitive computational performance characteristics of using multiple strides. Using a sliding window already significantly impacts the time to predict since we are running the predictions multiple times.

If this approach seems useful, a simple insight is that you need not pass the inputs in sequentially; instead, we typically build a batch with all of the windows and pass them through all at once.

Of course, there is a downside to this approach that might make you decide to choose to split the windows based on sentences or paragraphs. otherwise, your positional encoding will end up being "off" since you are sliding across the inputs.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .