Is it feasible to use GAN for high-quality image synthesis other than human faces?

Question

The famous Nvidia paper Progressive Growing of GANs for Improved Quality, Stability, and Variation, the GAN can generate hyperrealistic human faces. But, in the very same paper, images of other categories are rather disappointing and there hasn't seemed to be any improvements since then. Why is it the case? Is it because they didn't have enough training data for other categories? Or is it due to some fundamental limitation of GAN?

I have come across a paper talking about the limitations of GAN: Seeing What a GAN Cannot Generate.

Anybody using GAN for image synthesis other than human faces? Any success stories?

Answer 1

Generative Adversarial Networks, basically boil down to a combination of a generic Generator and a Discriminator trying to beat each other, so that the generator tries to generate much better images (usually from noise) and discriminator becomes much better at classification. So, no it is not just suited for only synthesis high quality human face synthesis but any image type.

In fact, not only can it be used for any high quality image synthesis, it can work on non-image data types as well (such as text, etc). It all depends on the type of neural network you are using for the discriminator and generator at the end of the day.

http://openaccess.thecvf.com/content_iccv_2017/html/Osokin_GANs_for_Biological_ICCV_2017_paper.html

The above refers to a paper synthesizing cell image through GANs, as I haven't personally used GANs on a practical level.

General blog explaining GANs:
https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/

Human faces are much more frequently tackled, for many reasons, usually that human faces are highly symmetric and have a lot of different features, usually more than other types of images, with the added difficulty of being that we as actual humans are usually good at recognizing faces - making a neural net to fool ourselves, makes it a challenging area of research.

Hope it helped! Do let me know if I'm wrong somewhere.

Answer 2

I'd challenge your assertion somewhat that the generated images of other categories are of much worse quality than the faces!

Take the bikes on transparent / solid backgrounds they look great!

Where the images fail a bit is with the more complex pictures which have a lot of elements where element bleed (covers bleeding into the floor, etc.) occurs. This is simply a result of the complexity of the image and the training data base.

As an example I have developed a GAN that generates "Vaporwave"-like imagery like this:

Now my results were generally poor because unlike faces my training set was highly diverse in terms of arrangements, elements, etc. If you look at the generated bed images in your example paper not only did the GAN have to learn and generate beds but also the highly complex backgrounds which did differ severely between training images whereas in the face example the image was zoomed in on the faces and obscuring the background.

If you use human faces in a normal background setting (e.g. with the scenery around them visible) your GAN will perform equally good or bad because there is so much more complexity to learn.

You can find my experiences with non-faces GAN on Kaggle but understand the bad results are mainly due to a very small training set and the fact that these images are very different (besides the color gradient which the GAN pics up very fast).

https://www.kaggle.com/fnguyen/vaporprogan