3
$\begingroup$

Attention has been used widely in recurrent networks to weight feature representations learned by the model. This is not a trivial task since recurrent networks have a hidden state that captures sequence information. The hidden state can be fed into a small MLP that produces a context vector summarizing the salient features of the hidden state.

In the context of NLP, convolutional networks are not as straightforward. They have the notion of channels that are different feature representations of the input, but are channels the equivalent to hidden states? Particularly, this raises two questions for me:

  • Why use attention in convolutional networks at all? Convolutions have shown to be adept feature detectors––for example, it is known that higher layers learn small features such as edges while lower layers learn more abstract representations. Would attention be used to sort through and weight these features?

  • In practice, how would attention be applied to convolutional networks? The output of these networks is usually (batch, channels, input_size) (at least in PyTorch), so how would the attention operations in recurrent networks be applied to the output of convolutional networks?


References

Convolutional Sequence to Sequence Learning, Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, Yann N. Dauphin, 2017

$\endgroup$
1
$\begingroup$

It seems that in the reference you provided they use attention to compute weights for the encoder representations based on the context provided by the decoder (Fig. 1). As far as I can tell, attention is applied after convolution (in fact, after the GLU step), so it does not affect the feature maps directly. Rather, attention is used to select the target words in the decoder. In other papers (e.g., this one), attention is applied directly to the feature maps in a way that is more similar to what you described.

Regarding your second question, the paper you reference actually provides a link to the source code they used. It is written in Lua (using Torch) instead of PyTorch, probably because PyTorch development was just starting when the paper was published. At any rate, you should be able to follow the Lua code and translate it into PyTorch.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.