How to build a commercial Image-Image search engine using LSH / Near Duplicate or some other algo on more than 20M images

TL;DR: HOW DO I APPLY LSH WITH A DEEP LEARNING MODEL TO BUILD A IMAGE-IMAGE SEARCH ENGINE ON >20M IMAGES?

I want to build a system where I am helping my colleagues to improve the search of incoming images. I'll be putting the pictures of how my data looks like at the end.

Problem: In the very broad terms, goal is to return the tok-k similar images from more than 20M images. My idea before was to get some similarity measure (l-1,l-2,cosine, SSIM etc etc) to get the similarity of two images OR to use a Siamese Network BUT , obviously, saving n*m dimensional vector of each image in memory is impossible and on top of that, comparison will take WHOLE LOT OF TIME when we are talking about live search obviously O(N) time.

So I stumbled across This OpenCV based search engine. And I get the idea that you decrease the dimensions to get a latent space, save it in memory, and then compare. But still, there 2 problems with that, I want to work with images that has text in them and obviously, comparing to let's say 20M is still a problem.

So I think I need to train a Neural Network for my images from scratch say ResNet or use existing weights.

So I got to know about LSH and Near-Duplicate algorithm where comparison is < O(N) for sure in worst case.

Now the problem is that I do not know how to use LSH with the CNN Neural Network. How do I train my model so that it can learn?

My idea for dimensionality reduction is that:

1. Train a siamese network with Triplet Loss which gives me 2 outputs and train it on similar and non similar images.
2. when the model is fully trained, get the low dimensional representations of all of the Images in memory (excluding the last layer)
3. Now when a new image comes, pass to the siamese network, get it's representation and compare it with the given representations.

HOW DO I APPLY LSH WITH THIS MODEL? How do I predict the top-k similar images because 20M images in memory is still not feasible. Please help. Any code, reference, video, tutorial will be helpful. (Not research paper)

• You don't need to use upper case letters if you're already making the sentence bold. Sometimes, upper case letters are used to "shout" or "scream" and people may not interpret them very well, so I suggest that you should avoid as much as possible using them.
– nbro
Dec 31 '20 at 14:07