0
$\begingroup$

I'm trying to design a system to optimize over a variable-length set (like a sentence) of variable length vectors (like words). But unlike a sentence, the order of words does not matter.

I'll have to make vector embeddings for my input vectors, so far is clear. And I'm familiar with RNN's power to make summaries of sentences and I'd like to make a summary of my set as well. The question is what tool is the best for this problem. Is there an alternative to RNN that can make a better summary of a variable-length set of embeddings? What is the best way to make a summary for my set?

$\endgroup$

1 Answer 1

0
$\begingroup$

One important part in this is, do you have any summary labels for your "sentences"? Otherwise it may be hard to say if your models to good summaries or not. If your data is purely numerical, it seems you are rather looking at a compression problem than a summarization problem.

As your sets lack structure, RNN will generalize poorly as those models are typically ordered. Instead you should be looking at Self Attention.

One approach is to view your vector sets as nodes in graphs (possibly unconnected). The technique SAGPool should generalize well to your case.

If you don't have summary labels you can set up a Graph Auto-Encoder using the SAGPool to train representations of your sets and treat the smallest layer as your summary.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .