Why don't people use nonlinear activation functions after projecting the query key value in attention?

It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.

This observation applies to the transformer, additive attention, etc.

  • $\begingroup$ I'm not sure if I got your question right, for the attention model where exactly would you place the non-linearity? Looking at Graph Attention Networks by Petar Velickovic, they do apply an activation function in eq. 5. $\endgroup$ – razvanc92 May 3 '19 at 7:21
  • 1
    $\begingroup$ Can you provide an example of someone not using nonlinear activations in their attention? $\endgroup$ – Philip Raeisghasem May 4 '19 at 21:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.