1
$\begingroup$

The original paper for style transfer only supports one image as an input. However I was interested if there is a model that uses multiple images, preferrably in a way that doesn't require users to provide their own image files, just requiring the model to be trained on images of a single style and then imitating that style without the original image files. Can this be achieved using variants of the algorithms of that paper or should other neural network types be used (eg. GANs, diffusion models...)

$\endgroup$
1
  • 1
    $\begingroup$ Nothing is stopping you from concatenating multiple images into a single one. Although it is a very bad solution, it is technically a workaround for using multiple images. The thing is that the solution in the paper wasn't designed for copying the style of multiple images, but for copying the style of a single image to another in a very fast way. What you're looking for is a latent space generative model. Stable Diffusion Img2Img, models like Pix2Pix, or others of this style can solve this problem. $\endgroup$
    – Cesar Ruiz
    Commented Jan 11 at 0:18

1 Answer 1

0
+50
$\begingroup$

For text-to-image Diffusion Models there are some popular customization techniques, like Textual Inversion, DreamBooth and Custom Diffusion.

Textual Inversion performs a few-shot finetuning of a text embedding $V^*$ of the model's text encoder, on a set of input images (usually 3 to 5). Specifically, after finetuning, style stransfer is performed by providing the following prompt, including the learned embedding $V^*$:

Painting of X in the style of $V^*$ art

DreamBooth uses a similar few-shot finetuning strategy, but it instead optimizes the whole model, obtaining better quality samples at the cost of memory-intensive finetuning and a heavy checkpoint (as it includes the whole model)

Custom Diffusion finetunes only a small subset of attention layers, reducing fine-tuning time and cost compared to DreamBooth, and allows the composition of multiple learned concepts, a setting where previous approaches struggled.

$\endgroup$

You must log in to answer this question.