5
$\begingroup$

image

These images are handmade, not auto-generated like they will be in production. Apologies for inaccuracies in the graph overlay.

I am trying to build an AI like that displayed in the diagram: when given a training set of images with their corresponding node maps of face/nose posture, and an image with a missing section (just a gap) with a node map, I would like it to reconstruct the initial image. My thoughts immediately went to GANs for this, but after some searching, the closest I could find were:

  • Face recreation without context/not filling gaps, just following pose (DeepFake)
  • Filling gaps in images, but with no node reference
  • Filling gaps from reference drawings/mappings, but with no way to provide sample images

I would like to hear about any implementations of such an algorithm, if possible optimised for faces, and if none exists, I would like to hear of how I would go about altering the generator of the GAN to work with the context/gap-fill bit (e.g a paper which talks about this idea, but doesn't implement it). Any guidance on the NN that is best for this type of task is also appreciated.

Improve this question
$\endgroup$
2
  • $\begingroup$ Not come across an exact implementation of this but would recommend you check out the generator architecture of StyleGAN by NVLabs. Its a typical GAN mapping from latents to an image distribution but thought it might be helpful because they have a trained Generator on faces. $\endgroup$
    – ashenoy
    Nov 16, 2019 at 4:57
  • $\begingroup$ @ashenoy thanks for your comment, and apologies for not getting back: I wanted to make sure I understood the paper before giving a response. StyleGAN seems interesting, and as though it would have potential. I was wondering, would it be possible to hijack StyleGAN's discriminator to take the nodes into account? I think it would probably be easier to alter the discriminator, rather than telling the generator the node positions explicitly. However, this then just leaves the question of how could we make it do gap-filling? We somehow need to feed it a 'prompt' - how could this be initialised? $\endgroup$
    – Geza Kerecsenyi
    Nov 20, 2019 at 20:10

1 Answer 1

Reset to default
1
$\begingroup$

I believe you may want to use a Sum Product Network for this task. SPNs are the state-of-the-art approach for face completion, and there are several more recent papers on this topic since the original above.

Importantly, the SPN paper also covers other approaches that work well for this task. If lower-resolution results are acceptable for your task, PCA with 100 or more components works surprisingly well. If your dataset is very large, a nearest-neighbor approach can work decently too.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your response. Please see my comment response to @ashenoy: could you please explain how to deal with some of the issues/concerns raised there about StyleGAN in terms of SPNs? $\endgroup$
    – Geza Kerecsenyi
    Nov 20, 2019 at 20:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .