Excuse me if you find this question too vague and not fitting to this forum and feel free to close it. The overall goal of my question is to get a better intuition of the attention concept and mechanism.
There is a high-level analogy between attention mechanisms (to be specific: in the transformer) and Google's PageRank algorithm: both claim and strive to calculate "relative importances" – of parts of a sentence or of web pages – without a thorough definition of what "importance" actually is. The meaning of "relative importance" as calculated by PageRank is intuitively clear even though it's recursive: the relative importance of a web page is the sum of the relative importances of the pages linking to it. (Graph-theoretically speaking, the relative importances are given by the eigenvector corresponding to the largest eigenvalue of the adjacency matrix.) The idea is, that when looking for web pages on a specific topic one should pay attention to the most "important" web pages (which PageRank helps to find).
I wonder if the high-level analogy can be put a bit deeper: How are – for example – the mathematics of attention mechanisms related to the mathematics of PageRank – if they are? Or is the analogy too superficial and misleading and should be forgotten?
Until now I could not develop an intuitive understanding what the relative importance of a token in a sentence is (on which attention then is focussed): important with respect to what? To other tokens or the sentence or even the "full model" as claimed here? Or isn't the goal of attention mechanisms better explained in terms of "what kinds of relations are there between the tokens in a sentence and between the tokens and the sentence as a whole, and how strong are they?" That's the background of my question.
Once again: excuse the vagueness and possibly confusion of this question, I'm aware of it.