Combinig output of two different machine learning models for accurate invoice data extraction: Is this a viable approach?

Question

I am working (trying to work) on a project to extract relevant information from invoices. Currently I don't achieve much good accuracy so am trying to come up with some new ideas. I am considering combining two machine learning models: Lilt and YOLO, but the specific models arent important, I want to ask you about my proposed workflow:

Initial Processing with Lilt: The invoice is first processed by the Lilt model and OCR engine to extract structured data based on the layout and content. Lilt's output is captured in a structured format, such as JSON.

Secondary Processing with YOLO: The same invoice is then processed by the YOLO model to detect and extract specific fields or text blocks (e.g., invoice number, date, total amount). YOLO's output is similarly captured in a structured format with confidence scores.

Comparison and Reconciliation: I will implement a function to compare the outputs from Lilt and YOLO.

For each field: If both models provide a value, I will compare the confidence scores and choose the higher one. If only one model provides a value, I will use that value.

My goal is to leverage the strengths of both models to achieve higher accuracy in extracting invoice data. However, I have some concerns and would appreciate feedback on the following points:

Integration Complexity: Are there best practices for integrating outputs from two different models effectively?

Performance: Will running two models sequentially significantly impact processing time, and how can this be optimized?

Data: The models would be trained not on same datasets. I am currently working with one czech invoice dataset in Layout/Lilt format that i had obtain but YOLO expect different input so i would need to annotate new dataset. Would that be a problem?

Accuracy: Is it common to combine output of several transformers?

Has anyone implemented a similar approach, and what lessons did you learn? Are there alternative methods that might achieve better results? Is my idea just too complicated or straight up useless?

Any insights, suggestions, or resources would be greatly appreciated!

Answer 1

I want to suggest you something. As you know, ensemble modeling is a great idea for good performance. Especially, you are trying to combine two different-architecture models. In this case, you have to evaluate the performance of each model against certain dataset. Then you can decide what to do next.