Should I use an unsupervised approach or train a classifier with many classes to build a deep image feature extractor?

Question

I'd like to build a deep feature extractor of images (using a Bi-linear CNN).

What would lead to the best results:

• an unsupervised approach (such as https://iopscience.iop.org/article/10.1088/1742-6596/1237/3/032044/meta), or
• training a classifier on as many classes as I can, and hope it generalized enough?

I would then like to use this extractor as:

• A weights initialization for other classical tasks
• A feature extractor for Few Shot Learning approaches

Answer 1

This is exactly the problem I am currently working on. I don't suggest that you use a supervised method to learn latent representation, as the model might learn shortcuts or only a most meaningful feature ignoring other latent features.

There are several self-supervised representation learning approaches:

1. SimCLR and SimCLR2
2. SwAV
3. SimSiam

Here is an overview of such approaches.

These approaches are aimed to learn a latent representation of images (pre-text task) that can be then used for any downstream tasks (classification, query similar samples). Here is an interesting article on this topic.

Another approach is to use generative models. The benefit of this approach is that you can also generate synthetic images from the embeddings that can help debug and evaluate your model.

One of the simplest architectures is autoencoders (VAE, VQ-VAE). Another option is generative adversarial networks. They are more difficult to train but can produce higher quality images than autoencoders. Another advantage is that the latent space is more disentangled meaning that each value might be semantically interpreted.

The main limitation is that the conventional GAN can produce samples only from random noise and to encode a real image the model has to be extended. There are different approaches to overcome this limitation, for example ALAE.

This paper compares some of these approaches.

While these approaches can easily capture coarse details such as hairstyle or background color, they are likely to ignore finer details. For medical imaging, the main goal is to identify cancer cells, which is essential for diagnosis. If the model does not learn this feature, it renders a latent representation of no avail.

However, latent space can be forced to take these features into account by guiding the model with a supervised downstream task such as classification (StylEX) or segmentation (EditGAN).