2
$\begingroup$

Are there some known neural networks that, given an input image, can generate a similar image, with the same topic?

Example: input = a photo of a cat on a green table, output = a generated photo of another cat on another green table.

Example 2: input = a portrait of a man with glasses and a beard, output = a portrait of a generated person with similar glasses / beard (see "ThisPersonDoesNotExist").

I imagine it is possible with a GAN, but more precisely which kind of architecture?

$\endgroup$

2 Answers 2

2
$\begingroup$

(Good) GANs are highly specific. If you had a specific domain in mind, e.g. just human faces then you could use something like a Bicycle GAN. On top of a Generator (& Discriminator) this includes an Encoder, which would take your image, encode it into a latent variable z, which you could then feed into the Generator, perhaps with extra noise to get variance. To my knowledge there don't exist any good pretrained variants so you would have to train it yourself.

On the other hand, I think that Diffusion models are better suited for this particular task, they are trained to denoise a noisy image. So all you would have to do is take your original, add (some) noise and then denoise it to get a similar image.

Both of these would produce an image similar not only in content but also in style, if you actually cared about variance in style and keeping only the content, I would perhaps suggest:

  • feeding the original image into a Captioning network
  • feeding the caption into a text-to-image Diffusion model
$\endgroup$
1
$\begingroup$

The closest literature to what you're suggesting is indeed related to GANs, specifically to arithmetic performed on the latent space learned by generators.

Check Unsupervised Representation Learning with deep convolutional generative adversarial networks.

The idea suggested in the paper is to "tune" specific characteristic of the generated images by discarding specific filters learned by the generator. For example, by removing filters related to specific objects like windows, we could generate images of rooms without windows (see second image below, top row: images generated with all filters, bottom row: images generated without window filters).

enter image description here top row: images generated with all filters, bottom row: images generated without window filters

Other papers applied and refined the same idea, showing that you can tune characteristic of faces like age/glasses and so on.

enter image description here

$\endgroup$

You must log in to answer this question.