1
$\begingroup$

I have this scenario where I need to measure the similarity between a 2d tensor t1: (100,8) and 61 tensors of the same shape(100,8). 100 represent time-steps and 8 is the no. of options.

I first tried flattening all tensors so that I can use the cosine similarity measure, but it slows down performance significantly.

I used pairwise comparisons without flattening like the approach below, but it made executing mathematical operations more complicated (e.g., diagonal operations): How to calculate a meaningful distance between multidimensional tensors.

What should I do to speed-up ops on flattened tensors/or are there alternatives to flattening?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I suggest you to flatten and use FAISS. I had the same problem but using faiss it was made simple it is creating indexing which is faster to compare. Using faiss indexing i am able to compare 1 vector with 21 million vector in 6 milliseconds.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for your comment. I tried FAISS based on you suggestion, but it does not seem to support incremental addition of training samples. Based on your experience, is there a possible workaround for such a scenario or should I look for something different? $\endgroup$
    – Vilainor
    Commented Sep 30, 2022 at 19:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .