# Does summing up word vectors destroy their meaning?

For example, I have a paragraph which I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform quantity.

One thing I've done is taken every word in the paragraph, vectorized it using GloVe word2vec and then summed up all of the vectors to create a "paragraph" vector, which I've then fed in as an input for my model. In doing so, have I destroyed any meaning the words might have possessed? Considering these two sentences would have the same vector: "My dog bit Dave" & "Dave bit my dog", how do I get around this? Am I approaching this wrong?

What other way can I train my model? If I take every word and feed that into my model, how do I know how many words I should take? How do I input these words? In the form of a 2D array, where each word vector is a column?

I want to be able to train a model that can classify text accurately. Surprisingly, I'm getting a high (>90%) for a relatively simple model like RandomForestClassifier just by using this summing up method. Any insights?

Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value.

To feed data with different lengths, you can also try padding and trimming it. Set a constant L for length of paragraph and trim/pad all list of word vectors to this length. Padding adds 0 vectors to the begining of the list and trimming trims the first part of a text until it is equal to L. Even in LSTM networks padding and trimming is still used as even though you can feed as long of a text to a LSTM as you want, you still have to process the word vectors in batches, which requires them to be the same length.

Example code in python on padding/trimming vector:

def pad_trim(list_vec, L):
//list vec: {vec1,vec2,vec3......}
//Assuming vecN have a size of 200
return list_vec[-L:0] if len(list_vec) > L else [[0] * 200] * (L-len(list_vec)) + list_vec


However in inference, you can ignore maximum length if you used a RNN based method, although as the network has not been trained on lengths more than L, it may perform better or worse.

Generally speaking, you should go for concatenating if possible, so you can keep all information in the sentence. However both may work just fine depending on your task.

For RNN based and CNN based model example, you should check this out: https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f

• So, what will my data be like? Each column of a 2D array is a word, and the number of columns is the arbitrary number of words I used? What model works best for such texts, then? Sounds like something a CNN would work on? – Arnav Das Jan 7 at 15:54
• Yes. The data will be a 2d array. CNN should work fine but for word based tasks, use LSTM for best performance, however you should test both to see which works better – Clement Hui Jan 7 at 15:58
• I used a simple RandomForestClassifier on the project I described in the other question and got an accuracy of 92%. Is this a good accuracy? Will turning that data into 2D and retraining a new type of neural network model improve the accuracy over a simpler model like RandomForestClassifier? (Generally speaking, of course.) – Arnav Das Jan 7 at 16:00
• 92% is actually really good accuracy, but I think that may be training accuracy, not testing accuracy. A model can easily overfits if you just sum up the word vectors, so maybe you should try testing it and see the accuracy – Clement Hui Jan 7 at 16:05
• Actually, I used sklearn and split my data into 2000 training and 300 testing. The 92% is on the testing set. – Arnav Das Jan 7 at 16:06

But because the inputs have to have a fixed length

Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph. This has the advantage of being able to take into account word order and constellations, as well as processing variable size inputs.

Intuitively, I would think simply adding word vectors will include a lot of noise from commonly occurring words that don't provide much meaningful information from the classification. I would consider Bayesian methods or dimensionality reduction methods to limit the input to the more useful input vectors.

• How can I use an LSTM to input paragraphs with variable lengths to classify them? – Arnav Das Jan 9 at 10:03