1
$\begingroup$

I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information.

I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a good generational model for GNN.

So I want to use Transformer. But then the challenge is how can I embed structural information there? Is there any open source artefact for Relational Transformer that I can use out of the box?

Any pointers?

Improve this question
$\endgroup$
5
  • $\begingroup$ can you clarify the "relational information"? $\endgroup$
    – CuCaRot
    Apr 4, 2022 at 1:30
  • $\begingroup$ Two nodes are connected by an edge with a relation. Think about a knowledge graph. $\endgroup$
    – Exploring
    Apr 5, 2022 at 9:59
  • $\begingroup$ Would it make sense to define a positional embedding that is a function of the distance between nodes in your graph representation? $\endgroup$
    – John St. John
    Apr 6, 2022 at 3:58
  • 1
    $\begingroup$ @JohnSt.John can you please explain in a bit more detail in an answer. Also can you please refer to an implementation if possible. $\endgroup$
    – Exploring
    Apr 6, 2022 at 4:09
  • $\begingroup$ Done, hopefully that was helpful! $\endgroup$
    – John St. John
    Apr 8, 2022 at 3:47

1 Answer 1

Reset to default
1
$\begingroup$

I am not totally sure that this solution would work for your problem, however I'll take a stab at it, and hopefully this helps!

One of my favorite resources so far for learning about transformers is this post http://nlp.seas.harvard.edu/2018/04/03/attention.html. I was able to follow along in a jupyter notebook and the only changes I remember I had to make to get it working were removing references to Variable. There is a section in there called Positional Encoding which may be relevant for what you want to do. Some people alternatively call this Positional Embedding. The choice of positional encoding is somewhat flexible. I have seen projects, such as Enformer from DeepMind (https://github.com/deepmind/deepmind-research/tree/master/enformer), which use custom position embeddings for the DNA sequence domain, which is centered in the middle of a sequence and extends out to the sides.

The above implementation/paper was very helpful for learning. Once I was ready to implement a transformer for a project I was working on, I found this implementation in the pytorch examples package a great place to start. In there you can see the class for the positional embedding and modify that as needed. https://github.com/pytorch/examples/tree/main/word_language_model

Now for your specific problem I did some googling for positional embedding/encoding and graph transformer, and I found a project that looks promising https://github.com/inria-thoth/GraphiT. Maybe that just does what you want, and their paper https://arxiv.org/abs/2106.05667 may have some useful background for you.

Hopefully these pointers help in your search!

Improve this answer
$\endgroup$
1
  • $\begingroup$ could there be different kind of edges using "Positional Embedding? $\endgroup$
    – Exploring
    Aug 4, 2022 at 0:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .