1
$\begingroup$

I'd like to build a program that would learn to automatically classify documents. The principle would be that, for each new document I add to the system, it would automatically infer in which category to classify the document. If it doesn't know, I would have to manually enter the category. For each hint I give to the system, the system would learn to refine its knowledge of document kinds. Something similar to face recognition in Picasa, but for documents.

More specifically, the documents would be invoices, and I want to classify them by vendors. Documents could be extracted as text, as image, or both.

Is there some know algorithms for this kind of job?

Up to now, I could think at two possible ways I could do it:

  • For images, I could add all the images of a given kind together, and record the pixels that are the most common to all images, to create a mask. For a new image, I would compare this mask with the image to determine how similar it is.
  • For text, I could record the list of words or sentences that are similar to all documents of a given kind.
  • Finally, I could do a combination of both techniques, for example by converting a PDF document to an image, or an image to text by OCR techniques.

I'm just wondering if I'm approaching the problem the right way. Especially about storing just enough information in the database.

$\endgroup$
3
$\begingroup$

Text approach:

Use LDA (Latent Dirichlet Allocation). LDA is unsupervised. Feed it in corpuses of text from the various documents (i.e. OCR them and feed LDA the results of OCR). It will then cluster them based on the contents of the text (with or without stop words - at your discretion). If possible, you could do a supervised approach of using a bag-of-words and any classifier such as an SVM or Random Forest.

Image Approach:

Use a CNN (convolutional neural network) and train it on images of the various vendors. If you don't have this class discrimination, and can't get it, then use an unsupervised approach such as an autoencoder and then cluster the points in the lower-dimensional autoencoder feature space.

$\endgroup$

protected by Community Apr 13 '17 at 11:51

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.