Are there any metrics suitable for evaluating conditional GANs in the context of medical imaging?

Question

I am currently training a conditional GAN for breast cancer images with three classes: benign, normal, and malignant. While researching evaluation methods, I found that GANs are commonly assessed using metrics like the Inception Score (IS) or the Fréchet Inception Distance (FID). However, as I understand it, these metrics rely on the pre-trained Inception V3 model, which was trained on natural images and may not perform well for medical images. This often results in low scores, even if the generated images are of good quality.

Given this limitation, are there any metrics suitable for evaluating conditional GANs in the context of medical imaging?

Ideally, I am looking for metrics that consider both the diversity of the generated images and their distinctiveness (similar to what the Inception Score measures).

Answer 1

Indeed instead of relying on pre-trained models from natural image datasets such as ImageNet or COCO-2014, you should train a classification model as feature extractor specifically on your concerned breast cancer images with three classes (benign, normal, malignant) to compute a domain specific FID which is more popular as image generative models' evaluation metric than IS.

If your dataset is not extremely large, start with a (ImageNet) pre-trained CNN model such as a ResNet which is an effective feature extractor and fine-tune it on your breast cancer dataset after preprocessing your images to a consistent size required by the chosen model. Then train the model to classify images into the above three classes by, say, the standard CE loss and save the trained model with the best performance of standard metrics like accuracy, F1-score, or ROC-AUC on the validation set. Finally extract feature embeddings from the penultimate layer of the trained model for both real and generated images to compute FID between the two types of embeddings via their respective mean and covariance statistics.

Answer 2

Most metrics for evaluating generative models, including IS and FID, are indeed based on pre-trained feature extractors. Regarding the data where these features extractors are trained, I agree with cinch's answer: for specific domains, it is good practice to take a pre-trained feature detector and fine-tune it to your data.

The way these feature extractors are trained has a major impact on the quality of the evaluation of generative models. For example, (1) found that Inception-based features are actually a bit lousy, and instead propose computing FID using embeddings from the CLIP (2) model, which in practice seem to be more informative. Similarly, other metrics such as (improved) Precision & Recall (3) and Density & Coverage (4) rely on embeddings from VGG-16 rather than InceptionV3.

Moving on from the feature detector and on to the metrics themselves: the IS is rarely used nowadays as it is too easily swayed by small image perturbations. Indeed, state-of-the-art GAN papers rarely report the IS since it was shown to really not provide useful insight into the quality of a generative model (5).

Speaking of Precision & Recall and Density & Coverage: precision/density approximate how much of the learned manifold overlaps with the data manifold (i.e., how much the generated data looks like the real data). On the other hand, recall/coverage try to measure how much of the real data distribution is captured by the generative model (i.e., how much of the data our model can generate).

Without getting into too many details (check the papers for those), Density & Coverage came as an "improvement" - or follow-up - to Precision & Recall: it is less sensitive to outliers and is more efficient to compute. However, it may still make sense to evaluate a model using P&R for at least these two reasons:

• It still seems to be more widely reported (popular?) than Density & Coverage, so for consistency in the literature it is still probably a good idea to report it.
• While Precision is bounded between 0 and 1, that is not the case for Density, which is unbounded above; therefore, you may argue that Precision is more easily interpretable.

Some other metrics that may be of interest to you:

• Class-Aware Fréchet Distance (6), which argues that the single-manifold Gaussian assumption made in FID may lose class information on labeled datasets. They use specialized feature detectors trained on specific domain data. However, this metric is for unconditional models.
• Intra-FID, first used in (7), reports the mean of the FIDs calculated separately for each class.
• Kernel Inception Distance (8), where the authors argue that FID can be biased when the data is small. The KID is unbiased by design. However, it is usually very correlated with FID and is not very commonly reported.

My advice: default to tracking FID (with CLIP features) and either P&R or D&C during training, and check within your specific literature niche if there are other relevant metrics. When reporting results, it's not a bad idea to evaluate your models on as many (within reason) metrics as possible, even if they end up in the appendix.

---

References:

(1) - Kynkäänniemi, T., Karras, T., Aittala, M., Aila, T., & Lehtinen, J. (2022). The role of imagenet classes in fréchet inception distance. arXiv preprint arXiv:2203.06026.

(2) - Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

(3) - Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., & Aila, T. (2019). Improved precision and recall metric for assessing generative models. Advances in neural information processing systems, 32.

(4) - Naeem, M. F., Oh, S. J., Uh, Y., Choi, Y., & Yoo, J. (2020, November). Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning (pp. 7176-7185). PMLR.

(5) - Barratt, S., & Sharma, R. (2018). A note on the inception score. arXiv preprint arXiv:1801.01973.

(6) - Liu, S., Wei, Y., Lu, J., & Zhou, J. (2018). An improved evaluation framework for generative adversarial networks. arXiv preprint arXiv:1803.07474.

(7) - Miyato, T., & Koyama, M. (2018). cGANs with projection discriminator. arXiv preprint arXiv:1802.05637.

(8) - Bińkowski, M., Sutherland, D. J., Arbel, M., & Gretton, A. (2018). Demystifying mmd gans. arXiv preprint arXiv:1801.01401.