TL;DR: HOW DO I APPLY
LSH WITH A DEEP LEARNING MODEL TO BUILD A IMAGE-IMAGE SEARCH ENGINE ON >20M IMAGES?
I want to build a system where I am helping my colleagues to improve the search of incoming images. I'll be putting the pictures of how my data looks like at the end.
Problem: In the very broad terms, goal is to return the
tok-k similar images from more than 20M images. My idea before was to get some
similarity measure (l-1,l-2,cosine, SSIM etc etc) to get the similarity of two images OR to use a
Siamese Network BUT , obviously, saving
n*m dimensional vector of each image in memory is impossible and on top of that, comparison will take WHOLE LOT OF TIME when we are talking about live search obviously
So I stumbled across This OpenCV based search engine. And I get the idea that you decrease the dimensions to get a latent space, save it in memory, and then compare. But still, there 2 problems with that, I want to work with images that has
text in them and obviously, comparing to let's say 20M is still a problem.
So I think I need to train a
Neural Network for my images from scratch say
ResNet or use existing weights.
So I got to know about
Near-Duplicate algorithm where comparison is <
O(N) for sure in worst case.
Now the problem is that I do not know how to use
LSH with the
CNN Neural Network. How do I train my model so that it can learn?
My idea for dimensionality reduction is that:
- Train a
Triplet Losswhich gives me 2 outputs and train it on similar and non similar images.
- when the model is fully trained, get the low dimensional representations of all of the Images in memory (excluding the last layer)
- Now when a new image comes, pass to the
siamesenetwork, get it's representation and compare it with the given representations.
HOW DO I APPLY
LSH WITH THIS MODEL? How do I predict the
top-k similar images because 20M images in memory is still not feasible. Please help. Any code, reference, video, tutorial will be helpful. (Not research paper)