Can one use an Artificial Neural Network to determine the size of an object in a photograph?

Question

My question relates to but doesn't duplicate a question that has been asked here.

I've Googled a lot for an answer to the question: Can you find the dimensions of an object in a photo if you don't know the distance between the lens and the object, and there are no "scales" in the image?

The overwhelming answer to this has been "no". This is, from my understanding, due to the fact that, in order to solve this problem with this equation,

$$Distance\ to\ object(mm) = \frac{f(mm) * real\ height(mm) * image\ height(pixels)}{object\ height(pixels) * sensor\ height(mm)} $$

you will need to know either the "real height" or the "distance to object". It's the age old issue of "two unknowns, one equation". That's unsolvable. A way around this is to place an object in the photo with a known dimension in the same plane as the unknown object, find the distance to this object and use that distance to calculate the size of the unknown (this relates to answer from the question I linked above). This is an equivalent of putting a ruler in the photo and it's a fine way to solve this problem easily.

This is where my question remains unanswered. What if there is no ruler? What if you want to find a way to solve the unsolvable problem? Can we train an Artificial Neural Network to approximate the value of the real height without the value of the object distance or use of a scale? Is there a way to leverage the unexpected solutions we can get from AI to solve a problem that is seemingly unsolvable?

Here is an example to solidify the nature of my question:

I would like to make an application where someone can pull out their phone, take a photo of a hail stone against the ground at a distance of ~1-3 ft, and have the application give them the hail stone dimensions. My project leader wants to make the application accessible, which means he doesn't want to force users to carry around a quarter or a special object of known dimensions to use as a scale.

In order to avoid the use of a scale, would it be possible to use all of the EXIF meta-data from these photos to train a neural network to approximate the size of the hail stone within a reasonable error tolerance? For some reason, I have it in my head that if there are enough relevant variables, we can design an ANN that can pick out some pattern to this problem that we humans are just unable to identify. Does anyone know if this is possible? If so, is there a deep learning model that can best suit this problem? If not, please put me out of my misery and tell me why it it's impossible.

Answer 1

In my thesis I actually solve the problem of depth estimation with a CNN based on a single monocular image so I can share my experiences for understanding that problem.

As you already stated in general you have the problem that you cannot recover the scale of the scene in an image by geometrical approaches directly. And that is still not the case even if you know the properties of your camera and lens, like focal length, but don't know any absolute sizes of the scene. However, a neural network is still able to solve the task of depth estimation based on monocular images (at least for fixed camera properties) due to known objects sizes it learned through the training on the dataset. That means it can use the learned size of specific objects and the relative depth relations to give a fairly good approximation of the depth in the scene.

However, in your special case this approach would not work, if I understand you correctly. If you just take a photo of a stone that can have arbitrary sizes and no depth cues or any unique patterns that relate to depth are present in the image there is no chance to ever estimate the absolute depth. A CNN eventually would probably just learn some average depth values or recurring depth patterns of your used dataset or memorize the whole training set to minimize the training error since it simply cannot solve this task. So you would not get a tool that does somehow generalize to new scenes. A neural network is still just a function approximator and not something magical that can solve the unsolveable.

For your usecase there could be some (complex) solutions that could give you a more or less accurate depth estimation. For example you could use a structure from motion approach where you somehow mesure the absolute camera movement with the accelerometer of the phone. Or best would be a stereo-camera based setup where you know the absolute camera displacement of the camera positions which could solve this task if you have textures in your images. With that you could find the absolute depth of specific points through a classical stereo depth estimation or by using a CNN that estimates the depth on the stereo image pair. Another approach would be to let the user input the phone height above the ground or approximate it through the accelerometer of the smartphone to then approximate the stone size based on the size in the image and the absolute known height above the ground (probably inaccurate).

Answer 2

Can one use an Artificial Neural Network to determine the size of an object in a photograph?

Yes: Learning Depth from Single Monocular Images

In the end, depth is just one special form of size.

Of course, you need something partially known, e.g. another car. You don't need to know the exact size of the car, but you know which size cars in general have. If you have an image without any reference, it is impossible.