3
$\begingroup$

With an RGB image of a paper sheet with text, I want to obtain an output image which is cropped and deskewed. Example of input:

enter image description here

I have tried non-AI tools (such as openCV.findContours) to find the 4 corners of the sheet, but it's not very robust in some lighting conditions, or if there are other elements on the photo.

So I see two options:

  • a NN with input=image, output=image, that does everything (including the deskewing, and even also the brightness adjustment). I'll just train it with thousands of images.

  • a NN with input=image, output=coordinates_of_4_corners. Then I'll do the cropping + deskewing with a homographic transform, and brightness adjustment with standard non-AI tools

Which approach would you use?

More generally what kind of architecture of neural network would you use in the general case input=image, output=image?

Is approach #2, for which input=image, output=coordinates possible? Or is there another segmentation method you would use here?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ have you tried preprocessing the image before applying findcontours? Like contrast boosting, conversion to hsv space, masking with edge detection. A neural net for such task sounds a real overkill to me, and complicate cause of the annotations that it would require. $\endgroup$
    – Edoardo Guerriero
    Mar 1, 2022 at 19:12
  • 1
    $\begingroup$ I agree that a neural net seems like overkill. That being said, this paper seems interesting: "Convolutional Neural Network Architecture for Geometric Matching" Ignacio Rocco, Relja Arandjelovic, Josef Sivic openaccess.thecvf.com/content_cvpr_2017/html/… $\endgroup$
    – The Guy with The Hat
    Mar 2, 2022 at 2:32

2 Answers 2

Reset to default
5
$\begingroup$

I think the second approach will be the best because it only requires that your training set is annotated with four labels for each of the four corners of the paper sheet.

This is sort of the idea of a Region Proposal Network which is used in Faster R-CNN (section 3.1).

Here is a reference implementation of a Region Proposal Network in PyTorch from the torchvision library. Notice how the network outputs boxes (in the forward() method) which is a tuple (x1, y1, x2, y2). From these four coordinates, you could crop the image to the desired paper sheet region.

Improve this answer
$\endgroup$
2
$\begingroup$

You could try U-Net for approach 1.

This is called the image-to-image translation problem in machine learning. You could see more architectures in this paper: https://arxiv.org/pdf/2101.08629.pdf

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ A U-net wouldn't help a whole lot if it's being spatially transformed, since the skip connections wouldn't line up correctly between the input and output. That said, I guess a U-net is probably better than an autoencoder-like model without any skip connections, so idk. $\endgroup$
    – The Guy with The Hat
    Mar 2, 2022 at 2:24
  • 1
    $\begingroup$ Thanks for your answer. $\endgroup$
    – logijaz
    Mar 3, 2022 at 8:55
  • 1
    $\begingroup$ @TheGuywithTheHat What kind of neural network would you use as a general tool for the general problem of image=in image=out, and you want the NN the learn transformations of images on thousands of examples? I would be interested in an answer about this, if you have one! Thanks $\endgroup$
    – logijaz
    Mar 3, 2022 at 8:56
  • 2
    $\begingroup$ @logijaz Accurate spatial transformations are something that current neural net architectures are just fundamentally unsuited to solve. U-nets connect pixels that are in the same location in the input and output, but with spatial transformations those pixels don't correspond to one another, so it's pointless. Other architectures can try to transform the image, but will lose a significant amount of detail (e.g. any text will become gibberish or disappear entirely). If you really want to involve a neural net, use it to find the corners, and then use traditional methods to transform the image. $\endgroup$
    – The Guy with The Hat
    Mar 3, 2022 at 22:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .