Most online sources recommend using versions of the Inception score to evaluate the synthetic images generated by a GAN. These scores are pre-trained on the InceptionV3 model. Does this mean that images need to have similar properties to those in ImageNet? My images have only one channel and are images of climate data so have very different properties. What is the best way to evaluate GAN-generated imagery for non-photographic data?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
6
Consider that the problem of evaluating (or measuring) the quality of generated images, can be expressed in terms of texture quality and/or perceptual quality.
- The texture quality measures how close the generated texture is to the original one. Intuitively, measuring textures is equal to measuring pixel-level differences. For example you can achieve this by a mean squared error (MSE) over the single pixels.
- Perceptual metrics look instead at the (global) meaning of the image rather than at its individual pixel values. Therefore, perceptual quality aims to measure the agreement in semantics of two images, ideally reflecting the human understanding of them.
Actually there is a trade-off between texture and perceptual quality, meaning that the same method can't maximize both. In general, VAEs are said to generate better looking textures while GANs to provide images with way better semantics: in other words, GANs generally generate plausible images but with inaccurate textures - indeed, for very big models the errors tend to narrow.
Does this mean that images need to have similar properties to those in ImageNet?
Well, the Inception score is defined from the activations of a pre-trained Inception-V3 so you're relying on what the model has learned. I mean it should be expected to better rank natural images rather than something that is radically different and never seen during training. As an alternative score you can look at the Frechet Inception Distance (FID) which also considers the distribution of the real images, which is expected to provide better evaluations in general.
What is the best way to evaluate GAN-generated imagery for non-photographic data?
It can be hard to answer but you can also try image processing metrics like the PSNR or the SSIM, as alternatives to IS and/or FID. These work for general kind of images, and you may want to try both in order to understand which better aligns with your notion of "better" quality. In general, the SSIM should be superior to both MSE and PSNR since it's designed to better match the human perception of images.
-
1$\begingroup$ Maybe it was implied in this answer, but one could in theory finetune (or train from scratch) an Inception or shallower model on some task (self-supervised task, if no label are available), and then use it to calculate IS. It can be expensive indeed, but it is still a solution. I do not also exclude that one could find a pretrained model on similar data, e.g satellite imagery. $\endgroup$– CiodarCommented Jun 23, 2023 at 13:11
-
$\begingroup$ @Ciodar, why do you need to fine-tune the Inception Model to calculate IS? I thought you could just plug in and calculate directly. Is it because there are objects in the dataset that are not in ImageNet ? Could you provide more clarification on this, thanks $\endgroup$– user77925Commented Mar 11 at 23:27
-
1$\begingroup$ @abcd The Inception Score measures the image quality by measuring the confidence and the diversity of the predictions computed by a separate, pre-trained classifier. The hypothesis is that if images look real, the classifier should classify them with high confidence. IF the domain of generated images differs from the classifier's domain, images could have a much lower confidence. Say you're generating images of a really different domain (e.g. satellite data, medical data, MNIST), a classifier trained on ImageNet should not have high confidence, even if images are qualitatively very good. $\endgroup$– CiodarCommented Mar 12 at 7:58
-
$\begingroup$ @Ciodar, Thanks for the detailed explanation. I have another question: I don't understand why research papers on GANs continue to use both Inception Score and FID. Since FID has shown improvement over Inception Score, why do people still use both instead of just using FID ? $\endgroup$– user77925Commented Mar 12 at 8:51
-
1$\begingroup$ @abcd I guess the main reason is to make sure their work is comparable with papers which relied only on IS $\endgroup$– CiodarCommented Mar 12 at 13:39