I asked ChatGPT to list some algorithms for identifying if one image is a cropped version of another or not. It suggested four algorithms that I know won't work, plus one that amounted to "train a neural network to do it".
Is this something a neural network is capable of doing? Would a neural network be able to abstract the general concept of "cropping", or would it simply learn to recognize cropped image pairs similar to those in the training data?
Is this something a neural network is capable of doing?
Theoretically, a neural network can estimate any function, so there is a chance that a neural network could do this if given the appropriate data. Given that cropped images would likely have highly variable resolution, I would say an architecture that can handle variable resolutions would be required. An architecture such as a fully convolution neural network would suffice.
Would a neural network be able to abstract the general concept of "cropping"
Let's consider some characteristics of cropped images
Can you come up with anymore? There could be some more abstract characteristics of cropping. But if given a dataset with a wide variety of images, that are randomly cropped or uncropped, a model could potentially learn some of these simple characteristics.
, or would it simply learn to recognize cropped image pairs similar to those in the training data?
*opinion: I can't image such a model would generalize very well on new data.
For example, consider a picture of half a tree. With enough training data, the model could potentially recognize what a tree is and the fact that only part of it is in the image and infer that it is cropped.
Hypothetically, if we take this a step further, and say the model can recognize that if an object in the foreground is not fully contained in the image, then the image is cropped. In this case, the model can generalize to new data, but only if there's a sufficiently large object in the foreground.
Now let's consider some potential failure modes, what if it's a picture of many trees? How can you tell that it's cropped? When you think about this, there's many cases where a model could fail, especially if it's a subtle crop. In the set of images below, the one on the left is the original, the middle is a subtle crop on all sides, and the right is a regular crop:
This idea could be tested on any image dataset, where images are either unchanged or cropped and resized on the fly with an appropriate label corresponding to uncropped or cropped. It would be interesting to see if the model is just obtaining performance via guessing or if it's actually learning characteristics of cropped images.
