Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures sequence information. The hidden state can be fed into a small MLP that produces a context vector summarizing the salient features of the hidden state.
In the context of NLP, convolutional networks are not as straightforward. They have the notion of channels that are different feature representations of the input, but are channels the equivalent to hidden states? Particularly, this raises two questions for me:
Why use attention in convolutional networks at all? Convolutions have shown to be adept feature detectors––for example, it is known that higher layers learn small features such as edges while lower layers learn more abstract representations. Would attention be used to sort through and weight these features?
In practice, how would attention be applied to convolutional networks? The output of these networks is usually
(batch, channels, input_size)(at least in PyTorch), so how would the attention operations in recurrent networks be applied to the output of convolutional networks?
Convolutional Sequence to Sequence Learning, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017