How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?

Question

I've been exploring the capabilities of the Diffusion based text-to-image models and am curious about its underlying mathematical framework. Specifically, I'm interested in understanding how the model handles the mapping from textual inputs to image outputs. Is this process considered a function, where each input is associated with a unique output, or is it a relation, allowing for multiple outputs from the same input due to the stochastic nature of the model? Where exactly the stochasticity lies in incase of relation and what is the nature of the relation ('many to many' or 'one to many')?

Consider the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et.al. as the underlying diffusion model for this question.

Answer 1

The paper you are referencing actually already achieving text-to-image generation.

It's a tricky question in my opinion, since the process that is modelled by neural network during denoising process is in fact a random stochastic process so there is no direct map between text embedding and resulting image. Actually diffusion process is a way to sample a "point" ( image in this case ) from conditional distribution (where the condition is a text input ).

But since neural networks are not stochastic in nature, the model itself is in fact a mapping in a sense. It's not bijective, so it's not a one-to-one map, but rather an injective map, so its many-to-one map. Since you can construct two inputs for simple feedforward layer such that it gives the same output. However it is not stochastic, that is if you give it the same inputs it will produce the same output every time.

The "randomness" or "generative" power of the models comes from the fact that each input is actually slightly different. Usually its common mechanism for all generative networks, the input variable is sampled from some known distribution ( often its normal gaussian ). So if you would give the model some prompt, to generate an image, something like the following will happened:

• Create embedding for the prompt ~ $x_{\text{text}}$
• Sample random variable $z \sim \mathcal{N}(0, 1)$.
• Concatenate them together to obtain the input $x = [x_{\text{text}}, z]$
• Run the neural network to create an associated image: $\text{DenoisingDiffusion}(x)$

So if in my toy example if $z$ is always the same value, the resulting image will actually be the same.

Answer 2

I'm only familiar with Stable Diffusion 1.x models, but I assume that all LDMs work fundamentally the same way. In my understanding the mapping from input text to output image is a deterministic function, when you take into account the used random seed. The starting state is determined by white noise in the latent space, which is then denoised and is conditioned with the input text.

Ah, vl_knd posted a good answer already.

Note that the parallel GPU implementation isn't 100% deterministic, even using the same seed you may get slightly different results. This happens because floating point operations aren't associative.

Edit: see some examples here.