1
$\begingroup$

With the help of artificial intelligence, it is possible to increase the resolution of images that are initially low resolution, bringing it to ultra-high resolution. Also, initially static images are turned into short animated videos. But I'm interested in AI upscaling of video materials such as old movies and perhaps cartoons.

In my opinion, the technology is like this:

  1. video file is dismantled, i.e. is decomposed into separate pictures-frames.
  2. using GAN, the resolution of each frame is brought to the required value.
  3. new frames are re-edited, i.e. new pictures-frames are combined into a new video file.
  4. as initial data for training the neural network, the original film or cartoon, images with similar characteristics, other films or cartoons, and a large group of high-resolution images similar to frames are used.

Do I understand the technology of AI upscaling of films and cartoons correctly, or are there some other nuances and important details that I forgot to mention? I would like to get a clearer.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

The approach you outline in the steps 1 to 4 will work to a degree, and may be used in some very basic video upscaling systems. However, the results may not be satisfcatory. A very typical problem will be flickering due to inconsistent upscaling choices made on consecutive frames. In addition, upscaling and interpolating the spatial dimensions of a video sequence will increase the pixel "jumps" between frames (more pixels will have large change between each frame), so to keep the output visually smooth it is often desirable to increase time resolution, i.e. increase the frame rate, interpolating smartly between frames.

The first problem, differences in details when processing frames separately, is called temporal coherence. There are a variety of ways to increase temporal coherence when upscaling. All require some comparison between consecutive frame images. Some are specified as soft constraints, or a loss metric applied to the processing during upscaling. Many models are trained specifically for video upscaling, taking multiple input frames, so that such losses can be used to improve the core model.

Sometimes, temporal coherence can be implemented as a modifier when recombining the upscaled frames - i.e. as part of your suggested step 3. This will be ok for simple interpolation upscaling that doesn't attempt to add missing details. It will tend towards smooth blurring for the upscales, similar to the simplest upscale options in image editors.

This is still an ongoing area of research, with papers routinely published. Here are two that I found on a quick search. I have not vetted these papers for whether they might be useful to you, they are just examples:

The second problem, frame rate increase, is solved by having models specifically built to solve that problem. Again this is usually achieved by having inputs of consecutive frames that need interpolating, but this time output will be an interstitial frame that is not an upscale of either input. Training data for these models is typically high frame rate videos where the input has had frames removed, and the output is the relevant missing frames.

Some video generation models perform both tasks - image and frame rate upscaling - starting with an initial small low frame rate generation, and alternating layers between frame rate increase and image size increase.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you! The “physics of the process” and related problems are clear to me. But the block diagrams and mathematics underlying the methods for increasing temporal coherence and increasing frame rates need to be studied separately. Because, as I understand it, GAN in its pure form is not used for this. Therefore, surely those models that solve both problems have a specific structure? $\endgroup$
    – ayr
    Commented Jan 17 at 14:56
  • $\begingroup$ Well, I think that using the keywords from your answer I will find enough information to study. $\endgroup$
    – ayr
    Commented Jan 17 at 14:58
  • 1
    $\begingroup$ @dtn: There are quite a few different approaches, and different kinds of models used to solve these problems. So there isn't a specific structure, it's more about how deep you want to look into the issue. Similar is true for image upscaling in the first place. A simple model is to linearly interpolate to get values for missing pixels. Slightly more sophisticated is to use bicubic interpolation. Wavelet decomposition. The modern AI-based upscalers don't do any of that, and even looking at the generative model upscalers, there are GAN-based, Diffusion-based etc. No single model or structure $\endgroup$
    – Neil Slater
    Commented Jan 17 at 15:29
  • $\begingroup$ In general, I would like to study the issue at such a level that I could experiment with the upscale of some old films, which are now perceived completely differently than before... Well, and get some results and practical knowledge. $\endgroup$
    – ayr
    Commented Jan 18 at 5:26
  • 1
    $\begingroup$ @dtn I suggest you start with a naive system based around your initial idea. Building it and experiencing the issue with temporal coherence and frame rate first hand will be valuable experience, and far more rewarding than trying to build the most sophisticated version you read about as the first attempt. Once you have a basic version, then you can refer to ideas in this answer to upgrade it $\endgroup$
    – Neil Slater
    Commented Jan 18 at 8:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .