1
$\begingroup$

I am quite new to text classification.

Using EAST text detection model, I get multiple strings that aren't words and most often have no meaning. For example, IDs, brand names, etc. I would like to classify them into two groups. Which models work the best and how should I preprocess the strings? I wanted to use Word2Vec, but I think it only works with real words and not with arbitrary strings.

| improve this question | |
$\endgroup$
1
$\begingroup$

I would just use a dictionary. A simple list lookup would tell you whether it's a recognised word or not. As an added bonus you can add some basic language processing, eg to identify inflected forms without listing them in your dictionary. Or use regular expressions to recognise ID numbers. ML is not really the right tool here.

| improve this answer | |
$\endgroup$
  • $\begingroup$ What do you mean with dictionary? the problem I have is I got strings such as „526//51“ which would be an ID I want to recognize. But also „A1213“ would be an ID. And unfortunately every ID just occurs once or twice. So I can‘t use stuff like tf-id $\endgroup$ – oezguensi Jun 25 '19 at 16:38
  • 1
    $\begingroup$ A dictionary is basically a word list. For IDs you could use regular expression matching, like /[A-Z][0-9]+/ to recognise a letter followed by one or more digits. You could even do it the other way round: use a regular expression to recognise words (basically /[a-z]+/, possibly with a hyphen included), and treat everything else as a non-word/ID. The matching words you can then check against a dictionary/list, either of words or (in order to exclude them) brand names. $\endgroup$ – Oliver Mason Jun 25 '19 at 17:37
  • $\begingroup$ I would like to use the power of labels and train a supervised model. What would be the best way to create embeddings. I read about different embedding methods such as hashing. $\endgroup$ – oezguensi Jun 25 '19 at 18:54
  • $\begingroup$ It's the wrong tool. $\endgroup$ – Oliver Mason Jun 26 '19 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.