Does summing up word vectors destroy their meaning?

Question

For example, I have a paragraph that I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform quantity.

One thing I've done is taken every word in the paragraph, vectorized it using GloVe word2vec, and then summed up all of the vectors to create a "paragraph" vector, which I've then fed in as an input for my model. In doing so, have I destroyed any meaning the words might have possessed?

Considering these two sentences would have the same vector:

My dog bit Dave

Dave bit my dog

How do I get around this? Am I approaching this wrong?

What other way can I train my model? If I take every word and feed that into my model, how do I know how many words I should take? How do I input these words? In the form of a 2D array, where each word vector is a column?

I want to be able to train a model that can classify text accurately. Surprisingly, I'm getting a high (>90%) for a relatively simple model like RandomForestClassifier just by using this summing up method. Any insights?

Answer 1

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value.

For details: https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d#ecd3

To feed data with different lengths, you can also try padding and trimming it. Set a constant L for length of paragraph and trim/pad all list of word vectors to this length. Padding adds 0 vectors to the begining of the list and trimming trims the first part of a text until it is equal to L. Even in LSTM networks padding and trimming is still used as even though you can feed as long of a text to a LSTM as you want, you still have to process the word vectors in batches, which requires them to be the same length.

Example code in python on padding/trimming vector:

def pad_trim(list_vec, L):
    //list vec: {vec1,vec2,vec3......}
    //Assuming vecN have a size of 200
    return list_vec[-L:0] if len(list_vec) > L else [[0] * 200] * (L-len(list_vec)) + list_vec

However in inference, you can ignore maximum length if you used a RNN based method, although as the network has not been trained on lengths more than L, it may perform better or worse.

Generally speaking, you should go for concatenating if possible, so you can keep all information in the sentence. However both may work just fine depending on your task.

For RNN based and CNN based model example, you should check this out: https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f

Answer 2

But because the inputs have to have a fixed length

Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph. This has the advantage of being able to take into account word order and constellations, as well as processing variable size inputs.

Intuitively, I would think simply adding word vectors will include a lot of noise from commonly occurring words that don't provide much meaningful information from the classification. I would consider Bayesian methods or dimensionality reduction methods to limit the input to the more useful input vectors.