Best approach for 2D Grid Image Segmentation

Question

I'm working on a project where I need to extract text from grocery discount flyers like the Costco announcement below (retrieved in a random google search, Costco is not the deal here):

If I just run OCR (like with Tesseract in python):

import cv2
import pytesseract
img = cv2.imread('costco.jpg')
text = pytesseract.image_to_string(img)
print(text)

I get:

Cadbury Chocolate

variety pack packet

ere $12.99 i rom hagst 31026 2012 > > Je laa > + a > > Wrigley’s Excel Gum variety > > Backol 24 > > $13.79 fom agus 26.202

OFF

Solon Extra virgin olive oil [...]

Which is a lot noisy.

My guess is that splitting the image to its base squares enchances the recognition.

However, I'm confused on how to do it. I can classify images using a CNN, but am not sure about object recognition.

Should I have a sliding window and train several "grid box" objects on a generic CNN and then provide this window data to be classified? How to adapt to distinct object window sizes?

Answer 1

This is a really cool problem. You already have a working model here are a few different ways of going forward with the project.

• Grouping text based on locality. "no segmentation" Text region extraction in a document image based on the Delaunay tessellation

• Segmentation Multiscale Edge-Based Text Extraction from Complex Images

• Training a map of the image then extracting text form the map. Multi Oriented Text Detection

Training the model to detect boxes followed by extracting the text from each box seems like a very smart direction to move in. This paper talks about the former. automatic image segmentation and edge detection Good luck!