0
$\begingroup$

In the recent preprint paper Tree-based Focused Web Crawling with Reinforcement Learning a new model is introduced to classify web pages called KwBiLSTM.

The input to this model is a featurized webpage which, later in the forward pass, is concatenated with a featurized version of a list of keywords. Together, these features are passed to the classification layer, which determines whether the webpage and the keywords correspond to the same topic (label 1 or 0).

From literature research I could not find any other work using this novel approach, namely that the topic being classified is supplied as an argument, rather than learned explicitly by providing a set of webpages labeled for a certain topic.

My question is: Are there other works using a similar topic-agnostic binary topic classifier?

$\endgroup$

1 Answer 1

1
$\begingroup$

I realized the problem can be solved as such:

Given a target text and a list of keywords, embed both using a sentence embedding model (such as MiniLM-L6-v2 from HuggingFace). The embeddings can then be compared using a similarity measure and classification can be done using a threshold value.

This approach is topic agnostic, since the list of key terms can be changed for any topic that needs to be found.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .