The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and GRUs.
However, having gone through the paper, as well as several online explanations, I still have trouble wrapping my head around how they work. How can a non-recurrent structure be able to deal with inputs of arbitrary length?
Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.
However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-attention, therefore, causes no influence. Let's say we use a transformer model with 512 limit of sequence length, then we pass a input sequence of 103 tokens. We padded it to 512 tokens. In the attention layer, positions from 104 to 512 are all masked, that is, they are not attending or being attended.
The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum sequence length supported by the transformer you have built.
We have found it useful to wrap our transformer in a class that allows us to programmatically use a sliding window across inputs that are longer than the supported transformer input length. If the input is less than or equal to the supported length, it is simply processed. If it is longer, we iteratively slide across the data, passing each window into the transformer and then aggregate the outputs.
When we take this approach, we do not typically slide the window one element (word embedding) at a time, but instead use a longer stride, usually two to five embeddings at a time. We have been intending to do some research into evaluating whether using multiple strides improves overall performance, but have not yet done so because of the prohibitive computational performance characteristics of using multiple strides. Using a sliding window already significantly impacts the time to predict since we are running the predictions multiple times.
If this approach seems useful, a simple insight is that you need not pass the inputs in sequentially; instead, we typically build a batch with all of the windows and pass them through all at once.
Of course, there is a downside to this approach that might make you decide to choose to split the windows based on sentences or paragraphs. otherwise, your positional encoding will end up being "off" since you are sliding across the inputs.
