# Code freezes and never returns when linear_kernel (sklearn.metrics.pairwise) is used on 20M Movielens dataset

I'm fairly new to ML/AI, i'm trying learn the content based recommendation - here is my source code - https://github.com/jaganlal/content-based-recommender

I'm using MovieLens 20M dataset - tags.csv to recommend similar movies based on its tag. But whenever i run (from sklearn.metrics.pairwise)

cosine_similarities = linear_kernel(tfidf_matrix, tfidf_matrix)

The python cell keeps executing and doesn't return at all. The code can be found at https://github.com/jaganlal/content-based-recommender

Am i doing anything wrong, any help is highly appreciated.

Thanks,

Jagan

## 1 Answer

A kernel is a precomputed distance function. This means that instead of computing the distances between pairs of points every time you need them, you'll just compute them all a single time, at the start, and cache the values in memory. This is a great idea if the dimensionality of your data is very high, and you expect to have to make a lot of distance computations. For example, if each vector was an high definition image with millions of pixels, computing the distance between two vectors would cost millions of operations.

A problem with this approach is that if you have many exemplars in your dataset, computing and storing the distances between every pair will become hugely expensive. In your case, you have 20 million exemplars. This means you have ~$$20,000,000^2$$ = $$4*10^14$$ pairs of distances to compute. If your computer does ~$$10^9$$ computations per second (as is normal these days), this will take you about $$10^5$$ seconds, or a bit more than a whole day. Much worse than this, you will need approximately 400TB of RAM to store the kernel. If you don't have this, your system will eventually start writing to disk, which will cause the calculation to take about 3 years instead of 1 day.

The solution here is not to do quadratic work with millions of datapoints. Either use a subset of the data, or try approximate distance methods.