Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?

Question

In section 2.1 of the Graph attention network paper

The graph attention layer is described as

as an initial step, a shared linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is applied to every node. We then perform self-attention on the nodes—a shared attentional mechanism a : RF ′ × RF ′→ R computes attention coefficients eij = a(Whi, Whj ) that indicate the importance of node j’s features to node i.

(forgive my amateur formatting)

The function a represents a fully connected neural network that takes the concatenated vector of Whi and Whj, then outputs a single value which is pushed through a softmax to get the attention score aij. Then, the embedding vectors Whj (for all neighboring nodes of i) are weighted and summed by the attention score as normal.

If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that? I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one, but I cannot comprehend how this works. Why can we use a shared matrix here and not there? Is it theoretical or technical?

Answer 1

To my understanding, there isn't any theoretical reason why the query, key and values weights are absent.

I feel that the difference may lie in the way the additive attention is calculated vs the dot-product one.

In the equations for the Graph Attention Network (GAT), there is for sure a difference between GAT and Transformers. However, I think you might be misunderstanding the relationship between the two. The attention mechanism in GATs is not equivalent to the one in Transformers. This means that one is not simply a rearrangement of the other; they are different inductive biases.

If my understanding is correct, this means that matrix W represents the query, the key and the value transformation matrices all in one. But how can it do that?

The equation states:

$$ e_{ij} = a(Wh_i,Wh_j ) $$

You are correct that a single weight matrix $W$ is used for the projection from the feature space to the feature map. However, this doesn't mean that $W$ represents the query, key, and value matrices, although it serves those roles if compared to the Transformer architecture.

I feel that you understand the forward pass of the GAT but lack intuition on why it is done like that. According to Aleks Gordic's commentary in the following video, he mentioned that he spoke with the authors of the paper. They said they had indeed tried using separate query, key, and value weight matrices but encountered overfitting in their experiments. They concluded that using a single weight matrix yielded more generalizable results. This makes sense when considering that the goal of GATs is not to construct a language model, which is a more complex problem. GATs most often operate on static graph data with features to learn. They are designed not to copy Transformers, but to adapt Transformer principles to graph data and objectives.

Is it theoretical or technical?

Based on what I understand, most inductive biases, if not all, are not theoretically grounded. If you think about it, there is no paper that says why one method theoretically performs better than another in neural networks. This is because it ultimately depends on your data and your final goal for the model. We only have experimental data to support our claims and a sense of intuition. The difficult part is to develop that intuition, which is based on previous experiments.

What I'm trying to say is that there is no reason why you couldn't try adding three different weight matrices instead of one. For the original idea of the Transformer, it makes sense to divide the attention into queries and keys because we are trying to simulate language, and language has a semantic ordering component. This means that it is not the same to say "Hello World" as it is to say "World Hello." Based on this, it is logical to have extra weights that can identify this difference.

For the datasets that were tried with GAT, they were citation networks and protein predictions. Predicting classes in citations may not require using more weights because the problem statement does not seem to require it. However, this is not set in stone. In the end, the only way to know if one architecture is better than another is through experimentation.