Why don't people use nonlinear activation functions after projecting the query key value in attention?

It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations.

This observation applies to the transformer, additive attention, etc.

  • $\begingroup$ I'm not sure if I got your question right, for the attention model where exactly would you place the non-linearity? Looking at Graph Attention Networks by Petar Velickovic, they do apply an activation function in eq. 5. $\endgroup$
    – razvanc92
    May 3, 2019 at 7:21
  • 1
    $\begingroup$ Can you provide an example of someone not using nonlinear activations in their attention? $\endgroup$ May 4, 2019 at 21:53
  • $\begingroup$ I think what he means is that the queries, keys and values are computed as linear projections, i.e. the input is simply multiplied by a matrix, q = x * W_q, k = x * W_k and v = x * W_v respectively. We could use a non-linear function on each of them, q = σ(x * W_q) etc., but it is redundant because later on we use the softmax function and at the end a MLP which also has non-linearities in it. $\endgroup$
    – Andreas K.
    Jul 23 at 8:13


You must log in to answer this question.

Browse other questions tagged .