As far as I understand, the attention model in a LLM is used to mitigate the vanishing gradient problem.

When using activation functions like the sigmoid function, deep neural networks may lead to gradients that are very close to zero (because you are multiplying tails that are close to zero), which in a way reduces the usefulness of these models.

However, couldn't you simply use a different activation function like ReLU to avoid this issue?

  • $\begingroup$ nobody stops you, but you loose the interpretation of attention (not anymore a distribution)… check “linear transformers” $\endgroup$
    – Alberto
    Commented May 5 at 16:44
  • $\begingroup$ @Alberto feel free to post an answer $\endgroup$
    – A. Darwin
    Commented May 6 at 19:40
  • $\begingroup$ Attention layer in Attention is All You Need's Transformer uses softmax, not the sigmoid function. Do you have something else in mind? $\endgroup$
    – pcpthm
    Commented May 7 at 12:16
  • $\begingroup$ @pcpthm I would say that with "like the sigmoid function" he meant "in that family of functions" $\endgroup$
    – Alberto
    Commented May 7 at 17:05

1 Answer 1


The attention module can be seen in many way. Indeed, you can just see as a way to construct a set of weights on the fly (in poor words this is what it does): $$ A = softmax(QK/\sqrt{d})\\ output = A\cdot V $$ Indeed, you can see it as a way to aggregate length-independent sequence (a dense layer does not have this property as it requires a fixed size input).
Such mechanisms can be found in literature as "hyper networks" (you use a network to estimate the weights of another network)

Nothing stops you from using a different activation function... whoever you will loose the "interpretation" of that matrix being an attention/distribution

What you gain instead is scalability. Indeed, the problem of the $O(n^2)$ complexity comes from the fact that you need the normalization of the softmax. However if you use another activation, as ReLU, or even linear, you can go back to the $O(n)$ beloved complexity


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .