Transformer architectures, based on the self-attention mechanism, have achieved outstanding performance in a variety of applications.
The main advantage of this approach is that the given token can interact with any token in the input sequence and extract global information since the first layer, whereas CNN has to stack multiple convolutional or pooling layers in order to achieve a receptive field, that would involve the whole input sequence.
By receptive field I mean the number of timestamps from the input signal on which does the output depend. For example, for sequence of two
kernel_size=3 receptive field is 5. And in transformer the output of the first blocks depends on the whole sequence.
However, this comes at large computational and memory cost in the vanilla formulation: $$ O(L^2) $$ where $L$ is the length of the sequence.
There have been proposed various mechanisms, that try to reduce this amount of computation:
- Random attention
- Window (Local attention)
- Global attention
All these forms of attention are illustrated below:
And one can combine different of these approaches as in the Big Bird paper
My question is about local attention, attending only to the tokens in the fixed neighborhood of size $K$. By doing so, one reduces the number of operations to: $$ O(L K) $$ However, now it is local as the ordinary convolution, and global receptive field will be achieved only via stacking many layers.
Are there any advantages of Local self-attention against CNN, or it can be beneficial only in combination with other forms of attention?