I am trying to make a personal ML project where my objective is using a photo from an invoice, for instance, a Walmart invoice, classify it as being a Walmart invoice and extract the total amount spent. I would then save this information in a relational database and infer some statistics about my spendings. The goal would be to classify invoices not only from Walmart but from the most frequent shops where I spend money and then extract the total amount spent. I already do this process manually, I insert my spendings in a relational database. I have a bunch of photos from different invoices that I have recorded over the past year for this purpose (training a model).

What algorithms would you guys recommend? From my point of view, I think that I need some natural language processing to extract the total amount spent and maybe a convolutional neural network to classify the invoice as being from a specific store?



There are three functional aspects to this project, each of which could be a sub-project, even though they are interrelated. The decoupling of these aspects in the associated R&D will likely improve the rate of development.

  1. Classify images — in this case, the vendor brand
  2. Locate and orient the text
  3. OCR the text — in this case, the total

These are certainly feasible in the particular case of invoices from vendors, so the entire project certainly is.

The goal is to produce two fields of information from input images.

  • Vendor code
  • Total in some specific monetary unit

That past invoices, the names of the vendors, and the totals of each invoice are available as training data may be helpful. Whether the volume of training data is sufficient can only be determined through a very complicated application of theory to some of the data metrics or by trial. It is recommended to do both.

What algorithms would you guys recommend?

It would be irresponsible to define an AI design in terms of what people recommend for algorithm. The algorithms that guys recommend are already covered by the path names for files in the algorithms directory paths, the example's directory paths, and the sections of the documentation of any good AI framework. Let's first talk about the design and then the algorithm options, so there is more of a basis for algorithm selection than the largely random reading selections of the members of a site.

The models involved and which can be parameterized such that they can be tuned through training are partly defined by the three sub-projects above.


For this project, there is no reason for the AI to recognize which text on the invoice is the vendor name. If there are a hundred vendor invoice types, the speed in which a single person can identify the key boundaries of each invoice type is orders of magnitude faster than the work necessary to develop a completely general algorithmic approach for automating that work. It would only be resource thrifty to develop such automation if there were thousands of vendors and constantly varying templates for invoices, which is likely in the case of personal finance.

These are the key bounds of the form quadrilaterals (four sided polygons) within which the following objects lie.

  • Vendor name text box
  • Total amount text box
  • Rectangle around the entire form

Digitizing the twelve points through one of the labelling programs is a model much more likely to produce a reliable system than the representation of three rectangular bounding boxes and tilts. This is because tilt angle requires at least two adjacent points anyway and the aspect ratio cannot be held constant in real scanning scenarios when the scanner might be replaced or produce different ratios with wear or the smoothness or wear characteristics of the paper invoice.

Doing all three things using a network would require more than one person's invoices for a decade, unless the person is a billionaire compulsive buyer with a team of secretaries doing scanning and data entry.

The models are then these.

  • Digitization of a quadrilateral document with varying contrast, brightness, tilt, horizontal pixels per inch, vertical pixels per inch, relative location and size of vendor name, and relative location and size of invoice total
  • Numeric characters and other monetary characters in a rectangular box
  • Brand image for the vendor, which may include a logo and type of arbitrary and possibly unique font

Now we can talk about artificial network approaches.

  • Adjusting for contrast, brightness, tilt, resolution, locations, and sizes will require a customized input, loss function, and layer arrangement based on the geometry involved. A GRU network may gain some advantages if the documents can be ordered chronologically, since the collection then becomes a time series, the trends of which can be exploited.
  • Monetary values can best be done using OCR libraries.
  • Recognizing the brand is probably best done with CNN as a categorization machine.

The system must indicate if the vendor and total are not found, in which case a new template is indicated, and the twelve points must be digitized for this new type.

It is possible to do this with one single deep convolutional network, but, again, the data set would need to be augmented. The one other way to do this is to create a filled-out-invoice generator to produce the volume of stochastically variable data across the various variability dimensions listed above to train the deep CNN.


Since many of these problems have been tackled earlier and we have quite a few good tools to handle images and text, the task does not seem to be so very difficult. But then you would only find out after actually trying out the solutions suggested.

I suggest the following approach:

  1. Use Tesseract and OpenCV to extract text from the images you have saved. You can refer to a good example of using tesseract with python here - https://www.pyimagesearch.com/2018/09/17/opencv-ocr-and-text-recognition-with-tesseract/

  2. Evaluate the results to plan and consider your next steps. Refer to the publications linked to the tutorials such as the EAST algorithm which describes the challenges and point to other studies on this topic. This will help you evaluate the other approaches which have been studied for different cases by other researchers.

  3. Assuming the text detection has worked well, proceed forward by detecting parts of the document by using row detection (find continuous white-space extending horizontally) and tab stop detection (continuous white-space vertically). Use the Tesseract/openCV detected text blocks for this purpose.

  4. Re-organize the detected text according to the layout identified by rows/columns. Next, recognize and discard the text you may not required, e.g. the logo, other associated details such as phone no of the store, tax breakup, change given, etc. These may be easily picked up by carefully crafted regular expressions.

  5. Run a spell checker to remove errors in recognition (aspell, pyenchant, etc.)

  6. Finally, from the results obtained, you could evaluate the remaining errors and fix them by either writing a set of regular expressions for easily identified OCR mistakes, or, else build a text model to learn and fix similar errors.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.