Bag of Tricks: n-grams as additional features?

Question

I've been playing with PyTorch's nn.EmbeddingBag for sentence classification for about a month. I've been doing some feature engineering, playing with different tokenizers, etc. I'm just trying to get the best performance out of this simple model as I can. I'm new to NLP, so I figured I should start small.

Today, by chance, I stumbled on this paper Bag of Tricks for Efficient Text Classification, which very well may be the inspiration for nn.EmbeddingBag. Regardless, I read the paper and saw that they increased performance through using "n-grams as additional features to capture some partial information about the local word order"

So by the wording of this sentence, specifically "additional features", I take it to mean that they made n-grams as part of their vocabulary. For example "abc news" is treated as a single word in the vocabulary, and then appended to the training data that is being embedded like so:

dataset = TextFromPandas(tweet_df)
label, sentence, ngrams = dataset[0]
label, sentence, ngrams

# out:

(1,
 'quake our deeds are the reason of this # earthquake may allah forgive us all',
 ['quake our',
  'our deeds',
  'deeds are',
  'are the',
  'the reason',
  'reason of',
  'of this',
  'this #',
  '# earthquake',
  'earthquake may',
  'may allah',
  'allah forgive',
  'forgive us',
  'us all'])

I just wanted to check my assumption, because the paper is not very explicit. I already tried to string n-grams together as a new sentence in place of the old, but performance dropped significantly.

I will continue to experiment, but I was wondering if anyone knows the specific mechanism?

Answer 1

Yes, N-grams is about joining $n$ words as one single token.

Keep in mind it will greatly increase your features size:

If you originally have 1000 unique words, notice you could get up to 1000² 2-gram (usually you don't get ALL the combinations, but notice the number of features can potentially grow huge!)

If your dataset contains between thousands or millions samples, it could be enough to train a simple bag-of-words. But when you use a bi-gram, you'll probably need at least a million samples and lot more training steps. Besides, you'll probably have to tweak the hyperparameters.

That's a common Machine Learning trade-off. A simple model is not very accurate. But a more complex model requires more data, more training and can overfit more easily.