Consider the following diagram from the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et. al.,

enter image description here

In the context of this diagram, I'm uncertain about the functionality of a particular component referred to as the "switch." Based on my understanding, the conditioning information always flows to the denoising step and is directed either to the cross-attention module or to concatenate with $z_{T}$, but not to both simultaneously. Is my understanding correct? Could you explain why the conditioning information cannot be passed to both components at the same time? Is there a specific reason or mechanism described in the paper for this design choice?

  • $\begingroup$ Good question. I also do not understand how the attention output is being used inside the UNET. $\endgroup$
    – Nathan G
    Commented Dec 24, 2023 at 22:33
  • $\begingroup$ I have only skimmed the paper so far, but my understanding is that you are correct: "not both simultaneously" $\endgroup$ Commented Mar 23 at 18:43
  • $\begingroup$ If the convolutional self-attention is implemented as discussed here, then perhaps the choice is between self-attention and cross attention. I can see why one couldn't do both. $\endgroup$ Commented Mar 23 at 20:21
  • $\begingroup$ Typo to correct: et. al. should be replaced with et al. et is a complete Latin word meaning "and", not an abbreviation. $\endgroup$ Commented May 1 at 18:41

1 Answer 1


Section 4.3.2 of the paper (on p. 7 in v2) answers this question:

By concatenating spatially aligned conditioning information to the input of $\epsilon_θ$, LDMs can serve as efficient general purpose image-to-image translation models. We use this to train models for semantic synthesis, super-resolution (Sec. 4.4) and inpainting (Sec. 4.5).

In other words, they switch the network based on the application. If the input is a class or text, they use the cross-attention mechanism. If the input is an image, they use the concatenation mechanism.

This makes a lot of sense. Textual and class guidance don't have spatial information, and thus cannot be concatenated to the spatial input to the denoising U-nets.

But class maps and images with to-be-inpainted regions are already spatial, and thus can be concatenated to the input to the U-Net. Cross attention would lose the spatial information these inputs provide.

This leads to a clear answer to your question: If you were guiding image generation both with text and with an existing image, you would likely use both paths.

  • $\begingroup$ Let's say there is an image captioned "a man riding a horse" pretrained in the network... the prompt asks for "an astronaut riding a horse", so the weights push the output towards "a man riding a horse" image, segment out "man" using the u-net, and replaces the cutout with an "astronaut" image pretrained in the network? But how does the posture and size of the "astronaut" on file (who is probably not in a horse-riding posture) be matched to the "man" who is in a horse-riding posture? $\endgroup$
    – James
    Commented May 2 at 2:06
  • $\begingroup$ @James That's a new question. Feel free to create a new question and link to it from here. $\endgroup$ Commented May 2 at 13:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .