I'd like to build a deep feature extractor of images (using a Bi-linear CNN).

What would lead to the best results:

I would then like to use this extractor as:

  • A weights initialization for other classical tasks
  • A feature extractor for Few Shot Learning approaches
  • 1
    $\begingroup$ Depends on what your downstream task is. There are so many ways to pretrain image feature extractors, and you will have to choose them specifically based on their performance on you desired downstream task. $\endgroup$
    – DKDK
    May 12, 2022 at 7:46
  • $\begingroup$ Please, edit your post to provide the details that are missing, which are mentioned in the previous task, i.e. what is/are your downstream task(s)? $\endgroup$
    – nbro
    May 13, 2022 at 9:09
  • $\begingroup$ Thank you for your answers, I've done so! $\endgroup$ May 13, 2022 at 9:21

1 Answer 1


This is exactly the problem I am currently working on. I don't suggest that you use a supervised method to learn latent representation, as the model might learn shortcuts or only a most meaningful feature ignoring other latent features.

There are several self-supervised representation learning approaches:

  1. SimCLR and SimCLR2
  2. SwAV
  3. SimSiam

Here is an overview of such approaches.

These approaches are aimed to learn a latent representation of images (pre-text task) that can be then used for any downstream tasks (classification, query similar samples). Here is an interesting article on this topic.

Another approach is to use generative models. The benefit of this approach is that you can also generate synthetic images from the embeddings that can help debug and evaluate your model.

One of the simplest architectures is autoencoders (VAE, VQ-VAE). Another option is generative adversarial networks. They are more difficult to train but can produce higher quality images than autoencoders. Another advantage is that the latent space is more disentangled meaning that each value might be semantically interpreted.

The main limitation is that the conventional GAN can produce samples only from random noise and to encode a real image the model has to be extended. There are different approaches to overcome this limitation, for example ALAE.

This paper compares some of these approaches.

While these approaches can easily capture coarse details such as hairstyle or background color, they are likely to ignore finer details. For medical imaging, the main goal is to identify cancer cells, which is essential for diagnosis. If the model does not learn this feature, it renders a latent representation of no avail.

However, latent space can be forced to take these features into account by guiding the model with a supervised downstream task such as classification (StylEX) or segmentation (EditGAN).

  • $\begingroup$ Thank you for the thorough comment! I'll accept it as the answer! $\endgroup$ May 13, 2022 at 14:42

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .