Questions tagged [linformer]

For questions about the Linformer, which was proposed in "Linformer: Self-Attention with Linear Complexity" (2020) by Sinong Wang et al.

Filter by
Sorted by
Tagged with
1 vote
0 answers
172 views

Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?

In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels ...
mshlis's user avatar
  • 2,359