Let's say that I want to classify whether a document is a legal document or not. I have a list of keywords that will be presented only in legal documents.
What is the proper way or algorithm to calculate probability based on this list?
1
-
4$\begingroup$ The way this question is posed, it seems trivial: if any/all (depending on interpretation) of the keywords are present in the document, it is a legal document. Else it is undetermined. If this is not what you intended, can you clarify your question? $\endgroup$– brazofuerteCommented Jan 4, 2021 at 21:14
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Maybe this is what you are looking for: https://en.wikipedia.org/wiki/Aho%E2%80%93Corasick_algorithm
Basically you would
- build and store a finite-state machine that resembles a trie with additional links between the various internal nodes using the given keywords.
- for the candidate document, go through it with the above finite-state machine.
Then based on the result
- If there is any match, then the document is valid.
- Otherwise it is possible that the document is invalid. However, in terms of the probability, I do not have any good idea yet.
$\begingroup$
$\endgroup$
This is a pretty classic case of document classification via the Naive-Bayes based Bag of Words Algorithm. It'll give you a prior probability based on your keywords. Try this link for more information. In your specific case, the "Bag" is already pre defined which makes things easier.
Another related question is this which may spawn more ideas for your use case. More generally though, if you format your problem as text classication using a feature vector constructed as some set of counts based on your keyword database then you can use pretty much any ML algorithm which gives you a baysian prior or some output probability.
