# Get Neural Network to predict a tag/class on a certain word using the surrounding words as context [PyTorch]?

I am somewhat a novice at the topic of Neural Netoworks and PyTorch.

I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).

For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):

        Sentence                    Label
here is a sentence for my network     N


sentence is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.

I have already gotten my dataset 'padded' out so they're all of equal size.

Any help is appreciated

• I don't know how to do this specifically with pytorch, but for a recent project, we used BERT to make embeddings for the sentences, and then used a k-neighbors classifier to classify the words. – Varun Vejalla Aug 8 '20 at 23:51
• @VarunVejalla So will that basically focus on my 'centre' word while taking the others into account? Thanks for the help also! – Justin Cunningham Aug 9 '20 at 1:06
• You're welcome! That's pretty much right. I explained more in my answer. – Varun Vejalla Aug 9 '20 at 2:02

## 1 Answer

You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words.

For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from $$0$$ to $$4$$). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.

There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.