1
$\begingroup$

I saw a video on Youtube about AI and Super Resolution Image Reconstruction with TecoGAN. I must say I am impressed.

Now, I am wondering how reliable this is.

I have learned at university that you lose information if you do not sample to fullfill Nyquist. I also don't think that the example images are in any way sparse...

Is the AI just trying to fill in the blanks by guessing?

This would be fine for entertainment, but probably not so much to enhance robbery pictures and charge people based on enhanced pictures. It also wouldn't be a good solution for improving the resolution of scientific data if it is just "guessing".

Improve this question
$\endgroup$
0

3 Answers 3

Reset to default
4
$\begingroup$

Yes, it's guessing. In the training phase, you show it lots of coarse and detailed pictures, and the algorithm learns a mapping from course to detailed. Then you present it a new coarse image, and it executes the same mapping. The information from the original picture is gone, and it cannot be retrieved, so it's filled in by analogy to other cases.

"Guessing" sounds a bit random, so it's more like a very informed guess. A bit like reading lots of books, and then being asked what word comes after "the cat sat on the" -- you're likely to say "mat", and will be right in many cases, but there's no guarantee that the most common word actually does occur. So now just substitute words with pixel values, and add a complex statistical model to make the decision, but you still won't know what the correct element is.

As you rightly say, this is fine for entertainment, but not for serious applications, where missing details in a crime scene are filled in according to how previous similar scenes may have looked.

Improve this answer
$\endgroup$
0
$\begingroup$

You actually don't have to loose information if you don't fulfill Nyquist — although that topic is quite advanced and has limitations. Still, super resolution is reliable and used by most 4K TVs today to upscale 1080p video to fit the 4K screen. You may notice TV ads for 4K TVs occasionally mentioning this.

What super resolution does is just generalising shapes. For example, imagine a simple image with just a black rectangle. You can easily image enlargening that image to whatever size you want because you know how it's supposed to look. By training a neural network on a lot of images it can learn to generalise features such as faces and enlarge them.

Of course you can't take a single pixel of a face and expand this into a full 100x100 image. Therefore, it's best to use super resolution to enlarge entire images, not just individual areas of the image. If you were to use it to enlarge a very small patch of text in the image, it may not replicate the text correctly and read as something else.

Furthermore, very good super resolution models are slow. Most also only increase the resolution of a single frame at a time. This means there might be inconstancies between sequential frames in a video.

Improve this answer
$\endgroup$
0
$\begingroup$

You were correct in that the model won't be able to reconstruct any missing information with complete certainty (an intuitively impossible task). As Oliver Mason mentioned, it is estimating what a similar image (from the training data it has been exposed to) would have looked like (I should note that in the vast majority of cases, we don't/can't actually know what the model's internal representation looks like or how it works, only how to get there - see explainable AI).

In other words, it attempts to match the image it is given to a "latent space" (different terms are sometimes used for the same concept, but you may find the links in the article helpful - also see this question) that it has approximated (representing the data it was trained on) and mapped to the output space (i.e., the "high-resolution" images).

In theory it is possible that AI could be used to enhance certain features of crime scene images (e.g., sharpening the contours of a face so a detective can better recognize the person) but again, this only approximates the "true" data. In practical settings, it is more likely that the raw data is fed into a model designed specifically for achieving some end goal (for example, facial recognition) and humans are taken out of the loop once a sufficiently accurate model is available.

Generally, "highly educated guessing" is an appropriate summary for this type of method and you are right in your assumption that this limits its usefulness in certain applications where precision is important. Though this is somewhat tangential, you might find machine learning-based computational fluid dynamics (e.g., this paper) enlightening; while most machine learning techniques (anything involving neural networks, which are inherently approximators and also often nondeterministic) cannot produce scientifically rigorous simulations in the way traditional algorithms might, they can speed up some of the processing/analysis in the way a cleverly devised heuristic might.

Here are a couple articles you might find informative (some of these are pretty sparse in information relating to machine learning, but most have links to in-depth treatments):

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .