I am somewhat a novice at the topic of Neural Netoworks and PyTorch.

I am trying to create a model that takes a word (that I have modified very slightly) and a 'window' of context around it and predicts one of 5 tags (the tags relate to what sort of action I should perform on that word to get its correct form).

For example, here's what I would call a window of size 7 and it's tag (what it means isn't too important, it's just the 'target'):

        Sentence                    Label
here is a sentence for my network     N

sentence is the word that I want the network to predict the label for, but the 3 words on either side provide contextual meaning. My problem is, how would I get a network to know I want it to predict for that central word but not outright ignore the others? I am familiar with more normal NLP tasks such as NMT and character level classification.

I have already gotten my dataset 'padded' out so they're all of equal size.

Any help is appreciated

  • $\begingroup$ I don't know how to do this specifically with pytorch, but for a recent project, we used BERT to make embeddings for the sentences, and then used a k-neighbors classifier to classify the words. $\endgroup$ Aug 8, 2020 at 23:51
  • $\begingroup$ @VarunVejalla So will that basically focus on my 'centre' word while taking the others into account? Thanks for the help also! $\endgroup$ Aug 9, 2020 at 1:06
  • $\begingroup$ You're welcome! That's pretty much right. I explained more in my answer. $\endgroup$ Aug 9, 2020 at 2:02

1 Answer 1


You may want to take a look at this article, but I'll summarize. You can use BERT (or some other tool) to make embeddings of every word in every sentence. Then for each word, make a contextualized embedding vector using the rest of the sentence. bert-embedding does all of this itself. Then keep the embedding vector for the important words.

For each important word, you would then have two pieces of information: the embedding vector and the correct label (which could easily be made into an integer from $0$ to $4$). Depending on the size of the embedding vectors, you could use PCA to reduce the size, although this may not be needed. Using this data, you can then train a neural network or use a k-nearest neighbors classifier.

There is more information in the article, which I suggest that you read. They do a better job explaining than me and also have some actual code you may want to look at.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .