1
$\begingroup$

In section 2.1 of the Graph attention network paper

The graph attention layer is described as

as an initial step, a shared linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is applied to every node. We then perform self-attention on the nodes—a shared attentional mechanism a : RF ′ × RF ′→ R computes attention coefficients eij = a(Whi, Whj ) that indicate the importance of node j’s features to node i.

(forgive my amateur formatting)

The function a represents a fully connected neural network that takes the concatenated vector of Whi and Whj, then outputs a single value which is pushed through a softmax to get the attention score aij. Then, the embedding vectors Whj (for all neighboring nodes of i) are weighted and summed by the attention score as normal.

If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that? I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one, but I cannot comprehend how this works. Why can we use a shared matrix here and not there? Is it theoretical or technical?

$\endgroup$

1 Answer 1

2
$\begingroup$

To my understanding, there isn't any theoretical reason why the query, key and values weights are absent.

I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one.

In the equations for the Graph Attention Network (GAT), there is for sure a difference between GAT and Transformers. However, I think you might be misunderstanding the relationship between the two. The attention mechanism in GATs is not equivalent to the one in Transformers. This means that one is not simply a rearrangement of the other; they are different inductive biases.

If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that?

The equation states:

$$ e_{ij} = a(Wh_i,Wh_j ) $$

You are correct that a single weight matrix $W$ is used for the projection from the feature space to the feature map. However, this doesn't mean that $W$ represents the query, key, and value matrices, although it serves those roles if compared to the Transformer architecture.

I feel that you understand the forward pass of the GAT but lack intuition on why it is done like that. According to Aleks Gordic's commentary in the following video, he mentioned that he spoke with the authors of the paper. They said they had indeed tried using separate query, key, and value weight matrices but encountered overfitting in their experiments. They concluded that using a single weight matrix yielded more generalizable results. This makes sense when considering that the goal of GATs is not to construct a language model, which is a more complex problem. GATs most often operate on static graph data with features to learn. They are designed not to copy Transformers, but to adapt Transformer principles to graph data and objectives.

Is it theoretical or technical?

Based on what I understand, most inductive biases, if not all, are not theoretically grounded. If you think about it, there is no paper that says why one method theoretically performs better than another in neural networks. This is because it ultimately depends on your data and your final goal for the model. We only have experimental data to support our claims and a sense of intuition. The difficult part is to develop that intuition, which is based on previous experiments.

What I'm trying to say is that there is no reason why you couldn't try adding three different weight matrices instead of one. For the original idea of the Transformer, it makes sense to divide the attention into queries and keys because we are trying to simulate language, and language has a semantic ordering component. This means that it is not the same to say "Hello World" as it is to say "World Hello." Based on this, it is logical to have extra weights that can identify this difference.

For the datasets that were tried with GAT, they were citation networks and protein predictions. Predicting classes in citations may not require using more weights because the problem statement does not seem to require it. However, this is not set in stone. In the end, the only way to know if one architecture is better than another is through experimentation.

$\endgroup$
8
  • $\begingroup$ Thank you for the explanation, this clears things up, especially with the fact that they actually tried to use different matrices first. $\endgroup$
    – oliver.c
    Commented Oct 3, 2023 at 6:30
  • $\begingroup$ Nice insight that a transformer-style attention in GAT causes overfitting. The following points confuse me a bit though: 1) In what way are transformer-attention and GAT-attention different inductive biases? 2) Inductive biases often are actually theoretically backed (s. Geometric Deep Learning). 3) You last point about the word ordering should be less related to the choice of attention but rather to the use of a positional encoding (PE), without PE, the attention is permutation insensitive. What do you think? $\endgroup$
    – Chillston
    Commented Oct 3, 2023 at 9:28
  • $\begingroup$ @Chillston 1) I define inductive biases as "a set of assumptions that the learner uses to predict outputs." GAT assumes that the data is a graph, and Transformers assume that the data is a complete graph. Using the same definition, we can say that our model assumes the data is simpler because we are using a single weight matrix. This is another reason why the models has differents inductive biases. $\endgroup$
    – Cesar Ruiz
    Commented Oct 4, 2023 at 19:19
  • 1
    $\begingroup$ Good point for 1), I definitely agree on that. For 2) I think there are a lot of logical arguments to say that a Transformer is more powerful in text processing than an RNN (e.g. Transformers are better scalable, use a more general relational inductive bias, ...). But I think there can be different opinions on this matter. Regarding 3): Transformer wo/ pos-encoding is permutation equivariant. It cannot use order information purley from the self-Attention and that's the reason why they put the pos-encoding (s. abstract here). $\endgroup$
    – Chillston
    Commented Oct 6, 2023 at 16:49
  • 1
    $\begingroup$ Also yea I can really recommend the Geometric Deep Learning stuff - it opens up a very nice perspective to think more generally about Deep Learning methods :) $\endgroup$
    – Chillston
    Commented Oct 6, 2023 at 16:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .