Get Neural Network to predict a tag/class on a certain word using the surrounding words as context [PyTorch]?

Question

I am somewhat a novice at the topic of Neural Netoworks and PyTorch.

I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).

For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):

        Sentence                    Label
here is a sentence for my network     N

sentence is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.

I have already gotten my dataset 'padded' out so they're all of equal size.

Any help is appreciated

Answer 1

You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words.

For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from $0$ to $4$). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.

There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.