4
$\begingroup$

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and GRUs.

However, having gone through the paper, as well as several online explanations, I still have trouble wrapping my head around how they work. How can a non-recurrent structure be able to deal with inputs of arbitrary length?

| improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Transformer cannot handle arbitrary length. You must have misunderstood something. For example, BERT is a transformer architecture (encoder only), and it has a maximum length of 512 tokens. A very useful blog for understanding transformers is The Illustrated Transformer $\endgroup$ – Astariul Aug 10 at 5:41
3
$\begingroup$

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.

However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-attention, therefore, causes no influence. Let's say we use a transformer model with 512 limit of sequence length, then we pass a input sequence of 103 tokens. We padded it to 512 tokens. In the attention layer, positions from 104 to 512 are all masked, that is, they are not attending or being attended.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer. Could you clarify one more point: What do you do when the input sequence is longer than 512/1024? Is a transformer simply inappropriate in that case? $\endgroup$ – chessprogrammer Aug 11 at 13:35
  • 2
    $\begingroup$ Yes, it's the limitation of BERT. Actually, there are tons of researches trying to solve this issue such as SpanBERT, Longformer, Reformer, Sparse transformer, etc. We could categorize them into two categories roughly. One is embracing the length limitation and divide the sequence into sections, while the others assume the attention could be sparsified. $\endgroup$ – tsu Aug 12 at 7:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.