Background, I have good understanding of ML 101 (supervised, unsupervised, tensorflow etc), however just getting into transformers & gen-AI.

I have recently started looking into Transformers/ViT and I'm trying to solve a problem, that can build a lookup for a given query(text) -> relevant features and also reverse mapping (given a image run similarity search on feature embedding vectors identify associated queries).

Images generally have a set of objects (although the main object features are pronounced in some or most cases), say our query=cat associated with a image (known mapping database) has not just cats, but humans, cars etc. The goal is to drop features (humans, cars etc) not related to the query and create lean feature representation for the query. Let's say we have (1-1k) instances of query=cat and different images associated with it, basically can the attention mechanism be used to discard unrelated features. (for example you have different breads of cats, different sizes, assume we could have some parameter to tune the depth of the features to retain, for example with few cat images and all capturing different aspects, the model may not have consensus on what feature to attention on, however in the case of images with good focus on cat but different breeds, sizes etc , we can say we can relate high level features, like contour, tail, four legs.( however there could be other features like hair patterns, coat color may be differing, so we can discard (retain feature-depth) by tuning our parameters (guessing no of encoder layers, or no of heads. etc)).

once we have the vector database, given a image/image features/patch embeddings, we would want to list all the associated queries using similarity search.

My understanding is pre transformers, we could (on a smaller scale) build the features using SIFT (local),HOG, ConvNet (large receptive field), run clustering to get feature centroids, build a histogram of feature-centroid to image-count from same class and retain feature-centroids (parameterized cut-off) that explains most images (run unseen good samples and adjust no of features to retain). this approach may run into compute issues (assuming you have 1-10 million different queries).

My question is, how can we accomplish it using ViT (or transformers in general)? given the general transformer architecture (encoder & decoder ) , can we use patch embeddings of images along with positional encoding, feed it on encoder side (for self attention) and on the decoder side pass query tokens ? (what are the scalability issues we can run into?). can we

I understand (image -> text models ) has ability to list tokens, (however those tokens are not normalized to the query categories we have). can we finetune/train any known existing models (10 million (query -> image) assume 1million distinct queries)

one use case I think of is in e-commerce, the images uploaded by a seller can be profiled/ranked for search.

I'm looking at the code here https://github.com/sheneman/ViT/blob/main/model.py and aim to run it on a smaller scale, if I can confirm the right approach.

I have may have made lot of assumptions, welcome any and all corrections.



You must log in to answer this question.