# Code freezes and never returns when linear_kernel (sklearn.metrics.pairwise) is used on 20M Movielens dataset

I'm fairly new to ML/AI, i'm trying learn the content based recommendation - here is my source code - https://github.com/jaganlal/content-based-recommender

I'm using MovieLens 20M dataset - tags.csv to recommend similar movies based on its tag. But whenever i run (from sklearn.metrics.pairwise)

cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)

The python cell keeps executing and doesn't return at all. The code can be found at https://github.com/jaganlal/content-based-recommender

Am i doing anything wrong, any help is highly appreciated.

Thanks,

Jagan

A problem with this approach is that if you have many exemplars in your dataset, computing and storing the distances between every pair will become hugely expensive. In your case, you have 20 million exemplars. This means you have ~$$20,000,000^2$$ = $$4*10^14$$ pairs of distances to compute. If your computer does ~$$10^9$$ computations per second (as is normal these days), this will take you about $$10^5$$ seconds, or a bit more than a whole day. Much worse than this, you will need approximately 400TB of RAM to store the kernel. If you don't have this, your system will eventually start writing to disk, which will cause the calculation to take about 3 years instead of 1 day.