I'm working on a content based recommendation engine for ebooks. I create document vectors with 300 features for every ebook using a word2vec model trained on google news and determine recommendations based on the closest vectors. So far I have run tests on a dataset consisting of 200 books from the gutenberg project in four different categories.

I find that a small group of books appears to be recommended a lot more then others. The most recommended book is recommended for over half the dataset. This book is Theaetus, in which the most commmon tokens are fairly specific to the book (tokens like socrates, theaetus). I can find no intuitive reason why this book would match so well with over half my dataset.

Is this common behavior when using document vectors to determine similarity? Are there any methods that would reduce this effect?

  • $\begingroup$ Would that be Theatetus has more occurrence than all other books? $\endgroup$
    – SmallChess
    Oct 9 '17 at 13:24
  • $\begingroup$ The model purely checks distances between vectors, so occurrence isn't really a factor here. $\endgroup$
    – Hartger
    Oct 9 '17 at 13:29

This issue was resolved by normalizing the vectors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.