4
$\begingroup$

I was looking at two papers

I'm trying to implement the second paper and I'm having some troubles understanding the differences between GAT and GaAN. By looking at equation 1 in GaAN paper I can see only two differences from GAT.

  • The first difference is that they are doing a dot product with the initial feature map and
  • Have another fully connected layer to project the result.

Is there something else that I'm missing?

|improve this question
$\endgroup$
  • $\begingroup$ wow the stuff is realy hardcore. About (1), looks like they are implementing it differently but still maintaing the structure, so its basicly the same. And (2) fc-layer is not important. But did you understand what do thay do further with input? to put a graph of any length they must do some aggregation over variable length and i $\endgroup$ – user8426627 Jun 18 '19 at 13:10
  • $\begingroup$ Regarding point 1 in the GaAN paper they are stating: "The difference between our aggregator and that in GAT is that we have adopted the key-value attention mechanism and the dot product attention while GAT does not compute additional value vectors and uses a fully-connected layer". From my understanding those two implementations are quite different. Lets try to break it down. Lets say we have 2 nodes i,j where each have an embedding size of 3. When we are computing the GAT aggregator we combine the two nodes embeddings and feed them to a single ff neural net so we are learning 6 parameters. $\endgroup$ – razvanc92 Jun 21 '19 at 13:48
  • $\begingroup$ When computing the GaAN aggregator we define two weight matrices of size [3,3]. We than multiply node i with the first one and node j with the second one and have two matrices of size [1,3]. Afterwards we can use the dot product between them so that we only have 1 (the attention score). Now looking at the number of parameters GAT uses 6 vs GaAN 18 I think we can say that the two implementations are not similar. Please correct me if I am wrong. $\endgroup$ – razvanc92 Jun 21 '19 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.