1
$\begingroup$

In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels like it's in equation 20 utilizing JL, and I do not see where characteristics of self-attention come into play).

What am I missing?

enter image description here

Improve this question
$\endgroup$

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .