2
$\begingroup$

I'm not very knowledgeable in this field but I'm wondering if any research or information already exists for the following situation:

I have some data that may or may not look similar to each other. Each represents a node that represents a vector of size 128.

similar vectors

And they are inserted into a tree graph according to similarity.

better graph

and a new node is created with an edge connecting the most similar vertex node found in the entire tree graph.

Except I'm wasting a lot of time searching through the entire graph to insert each new node when I could narrow down my search according to previous information. Imagine a trunk node saying "Oh, I saw a node like you before, it went down this branch. And don't bother going down that other branch." I could reduce the cost of searching the entire tree structure if there was a clever way to remember if a similar node went down a certain path.

I've thought about some ways to use caching or creating a look-up table but these are very memory intensive methods and will become slower the longer the program runs on. I have some other ideas I am playing around with but I was hoping someone could point me in the right direction before I started trying out weird ideas.

Edit: added a better (more realistic) graph picture

Improve this question
$\endgroup$
2
  • $\begingroup$ might be late, but if your similarity metric abide the triangle inequality $d(a,c) \leq d(a,b) + d(b,c)$, you can use M-Tree (metric tree) to store/retrieve the node more efficiently. You can also query K-closest node by using M-Tree. $\endgroup$
    – Sanyou
    Commented Oct 5, 2021 at 5:40
  • 1
    $\begingroup$ Thank you for answering! That's a great idea to explore, as I haven't seen a good solution to tackle this problem yet $\endgroup$
    – JungleBird
    Commented Oct 6, 2021 at 22:58

1 Answer 1

Reset to default
0
$\begingroup$

It sounds like you may be looking for the A* search algorithm. It is a search algorithm, like DFS and BFS, but it will explore only the most promising branches based on a heuristic function you supply. The difficult part of implementing this is deciding on a low-cost, admissible heuristic.

Excited to reflect nbro's suggestion from comments.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ A* does not really prune branches, but it explores branches according to the admissible heuristic, so, more intuitively, we could say that it explores branches based on how promising they are, but it doesn't really discard any branch. I'm not sure if A* is really the solution the problem. $\endgroup$
    – nbro
    Commented Jan 4, 2021 at 21:57
  • 1
    $\begingroup$ I would say that within the search space branches are "pruned" if they are never explored. Is there other terminology for that you would look for? $\endgroup$
    – Alina Barnett
    Commented Jan 5, 2021 at 16:17
  • $\begingroup$ The reason why I think "prune" may not be the most appropriate term here is that, if there's no goal and the graph is finite, for instance, A* should explore all nodes, so, in that case, it doesn't prune anything. However, you're right that it may not visit certain parts of the search space if all paths that lead to the goal from a start node have already been explored, so, in those cases, we can think of certain parts of the search space being pruned. So, I was a bit imprecise in my previous comment, so your description was right (but not applicable in all cases). $\endgroup$
    – nbro
    Commented Jan 5, 2021 at 16:22
  • $\begingroup$ In any case, I'm still not sure whether A* is the right tool to solve OP's problem. Maybe you should try to explain more in detail how A* should be applied in the OP's case. $\endgroup$
    – nbro
    Commented Jan 5, 2021 at 16:27
  • 1
    $\begingroup$ Thank you guys for answering my question! I didn't know that kind of search algorithm was called the A* search algorithm. But nbro has a point since it seems like the A* algorithm requires a destination to work towards. This problem is a real head-scratcher because the data isn't discrete; it's like drawing lines in the sand to mark how far the waves reach into the beach. Using a heuristic has it's merits but it breaks down when that heuristic comes down to a razor's margin at a fork in the road. $\endgroup$
    – JungleBird
    Commented Jan 6, 2021 at 5:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .