The main advantages of the self-attention mechanism are:

  • Ability to capture long-range dependencies
  • Ease to parallelize on GPU or TPU

However, I wonder why the same goals cannot be achieved by global depthwise convolution (with the kernel size equal to the length of the input sequence) with a comparable amount of flops.


In the following, I am comparing against the original architecture from the paper Attention Is All You Need.


Consider the depthwise convolution of size $L$ with circular padding: $$ y_{t,c} = W_{t^{'},c} x_{t^{'} + t, c} $$ Here, $x$ is the input signal and $y$ is the output signal, $t$ is the position in the sequence, and $c$ is the channel index. Since the convolution is depthwise the given output channel depends on the unique input channel (we would like to have linear complexity in the dimension of the embedding vector).

After a single convolution, one definitely would not have any interactions between the tokens in the sequence.

However, a two-layer convolutional network with these tokens is able to capture long-range pair-wise interactions: $$ x_{t,c}^{(2)} = W_{t^{''},c}^{(2)} \sigma(W_{t^{'},c}^{(1)} x_{t^{'} + t, c}^{(0)}) $$

And by stacking a not very large number of these layers (like 12 or 24) one can model interactions between tokens in the sequence of arbitrary complexity.

Comparison of complexity:

The asymptotic complexity of both approaches seems to be the same.

  • Attention: $O (L^2 d)$

  • Depthwise convolution: $O (L^2 d)$

However, dot product attention seems to be a rather intuitive and biologically motivated operation that is crucial for sequence problems.

Has this question been studied in the literature or discussed somewhere before?


De-facto global depthwise convolution is used in MLP-Mixer. One stage performs convolution with global receptive field (of the size of feature map), and other operation is pointwise convolution with kernel_size=1.

enter image description here

  • $\begingroup$ This is an old question, but can you please provide a source that describes this "global depthwise convolution"? $\endgroup$
    – nbro
    Dec 5 '21 at 16:44
  • 1
    $\begingroup$ @nbro ok, I'll add information $\endgroup$ Dec 5 '21 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.