Are there any advantages of the local attention against convolutions?

Question

Transformer architectures, based on the self-attention mechanism, have achieved outstanding performance in a variety of applications.

The main advantage of this approach is that the given token can interact with any token in the input sequence and extract global information since the first layer, whereas CNN has to stack multiple convolutional or pooling layers in order to achieve a receptive field, that would involve the whole input sequence.

By receptive field I mean the number of timestamps from the input signal on which does the output depend. For example, for sequence of two Conv1D with kernel_size=3 receptive field is 5. And in transformer the output of the first blocks depends on the whole sequence.

However, this comes at large computational and memory cost in the vanilla formulation: $$ O(L^2) $$ where $L$ is the length of the sequence.

There have been proposed various mechanisms, that try to reduce this amount of computation:

• Random attention
• Window (Local attention)
• Global attention

All these forms of attention are illustrated below:

And one can combine different of these approaches as in the Big Bird paper

My question is about local attention, attending only to the tokens in the fixed neighborhood of size $K$. By doing so, one reduces the number of operations to: $$ O(L K) $$ However, now it is local as the ordinary convolution, and global receptive field will be achieved only via stacking many layers.

Are there any advantages of Local self-attention against CNN, or it can be beneficial only in combination with other forms of attention?

Answer 1

It is true that when using local attention with a window of size 5, the "receptive field" is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a key difference in how the learned weights are applied to the inputs.

In a CNN, the values of the many convolutional kernels are learned, but once learned, the kernels are static. In other words, at every position in the input (whether it be a 1D signal or 2D image), the dot product between the inputs within the window and the same CNN kernels is taken, and then a non-linear function applied.

With attention, the Query/Key/Value matrices additionally allow context to be taken into account. Instead of taking the dot-product of the input region with a set of fixed kernels, the additional matrices are effectively used to dynamically compute a new set of kernels for each position. "Attention" basically figures out for each convolution, which inputs are important (which inputs the network should "pay attention" to) by computing higher-valued weights using Q, K, and V.

I highly recommend reading a breakdown of the original "Attention is All You Need" paper such as this blog post: https://jalammar.github.io/illustrated-transformer/