0
$\begingroup$

I know convolutional neural networks are commonly used for image recognition, but I was wondering if they would be able to distinguish between predominantly text-based documents vs something like objects. For example, if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image? If not, is there a different AI technique better suited that is purely image-based, or would it require OCR and leveraging the text in the invoice?

Update: based on a comment, my ask my not be clear. I'm not trying to see if the CNN can differentiate between a document (mostly text based image) and a photo image. I want to know if based on a gif/jpeg/png of a document (no OCR performed) a CNN would be able to classify the documents, which basically could be used as a means of identifying the vendor.

Improve this question
$\endgroup$
4
  • $\begingroup$ When you say "be able to distinguish between predominantly text-based documents vs something like objects", are you asking if CNNs would be able to distinguish between 1. images that contain predominantly text and 2. images that contain predominantly objects? Then you say "if you trained using images of the first page of invoices matched to a vendor name, could you get a CNN to predict the vendor based on an image?", here "first page of invoices matched to a vendor name", I assume that invoices are examples of documents that contain mainly text... $\endgroup$
    – nbro
    Dec 13, 2020 at 12:20
  • $\begingroup$ but then you say "to predict the vendor based on an image", here "image" refers to the invoice, right? So, are you asking whether CNNs can actually perform classification of text-based documents? $\endgroup$
    – nbro
    Dec 13, 2020 at 12:21
  • $\begingroup$ @nbro Yes, I'm asking if they can perform classification of text-based documents, but without relying on OCR. So basically, classify based on how the document "looks". So, there's the same logo top left, similar position of text, etc. Granted, not two invoices will be the same, date will change, etc. Just wondering if a predominantly text based image provides enough nuance for a CNN to classify or not. $\endgroup$
    – Rocket04
    Dec 14, 2020 at 14:17
  • $\begingroup$ I will clarify the question. $\endgroup$
    – Rocket04
    Dec 14, 2020 at 14:17

1 Answer 1

Reset to default
0
$\begingroup$

Yes, it is possible, CNNs can also be used for OCR (see the MNIST task and this blog), although it's not the common way for OCR because it is considered a bit overkill and inefficient. Furthermore, OCR is considered a solved problem already and don't need deep learning to do well, unless perhaps under unfavorable conditions like dark lighting, complex background of the document, weird fonts, occlusion of text, etc.

Breaking down your questions:

  1. a CNN can likely classify an image as "this image has a word document in it" vs "this image does not have a word document in it". The word document itself, and the outline of the words, should form distinct enough features for the CNN to discriminate against
  2. a CNN can likely identify the vendor based on the logo of the vendor if you have enough training images of it at various distortions (angles, occlusions, different lighting, etc)
  3. a CNN can likely do OCR well enough to extract the text, and then an entity extraction model can identify the vendor from the words. However, this approach would be uncommon as OCR is already considered a solved problem. But a neural network can potentially help preprocessing difficult OCR images (rotating image, identifying text boundaries, fix lighting, perform super-resolution, etc) to increase OCR accuracy
Improve this answer
$\endgroup$
2
  • $\begingroup$ What I'd like to do is identify the vendor purely based on the image, not using any OCR. So I'd have folders for vendor1, vendor2, vendor3, etc. with the first page of dozens of invoices for each to train the CNN. Just not sure if an image of a document has enough nuances for the CNN to distinguish, especially since these are not different document types, they're all invoices, just different vendors. $\endgroup$
    – Rocket04
    Dec 17, 2020 at 17:26
  • $\begingroup$ It's theoretically possible for CNN to do well, but I think in practice with commodity hardware and the CNN architecture we have today, it will perform poorly. Would be better to use OCR, then use fuzzy keyword matching to see if the receipt mentions a vendor names. You'd only need to build a database of vendor names with this method, which is much easier than designing + training a neural network model. $\endgroup$
    – user3667125
    Dec 18, 2020 at 22:43

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .