Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [diffusion-models]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
20 views

Minimally reproducible diffusion model

I am working on my thesis where I test the speed of sampling from probability distributions of different algorithms. I also want to integrate the sampling algorithms into some machine learning models ...
stefan stojkovikj's user avatar
1 vote
0 answers
22 views

What are all the inputs that support diversity of images in text to image generation?

For this question, consider the stable diffusion model. For a given text embedding, Stable Diffusion can generate diverse images. In this context, 'diversity' refers to the variation among the images ...
hanugm's user avatar
  • 3,890
1 vote
0 answers
21 views

How does this distribution change in "Understanding Diffusion Models: A Unified Perspective"?

In the paper Understanding Diffusion Models: A Unified Perspective, how did we go from equation $(44)$ to $(45)$? I couldn't find the details in the paper. How does the distribtuion for, the ...
Harry's user avatar
  • 11
0 votes
0 answers
10 views

Synthesising Images from Features

I'm currently trying to understand image generation a bit better. I'm working on a DDPM to generate samples from the MNIST set. My question doesn't really have anything to do with that, it's more just ...
euleriwt's user avatar
0 votes
0 answers
23 views

Confusion on forward process of diffusion models

I'm reading Chp 20 of Deep Learning Foundations and Concepts and am having trouble understanding how these two equations are derived. For context $$ (1) \ \ q(z_t|z_{t-1})=\mathscr{N}(z_t|\sqrt{\...
Alejandro Espino's user avatar
0 votes
0 answers
16 views

How are conditional models different from supervised models?

I'm wondering what the difference between conditional learning and supervised learning is - especially in diffusion models? Am I correct to assume that diffusion models are supervised because in ...
euleriwt's user avatar
0 votes
0 answers
14 views

Why does UNet often output noisy pattern in blank/homogeneous region?

I am recently implementing DDPM model from scratch, and I discovered that UNet often tends to give noisy output in blank region. Here is an example with FashionMNIST, my DDPM seems to generate OK ...
Dibbla's user avatar
  • 11
1 vote
1 answer
142 views

Trying to understand some derivation in the paper: Deep Unsupervised Learning using Nonequilibrium Thermodynamics

I have recently been learning about diffusion models and trying to derive all the results in the paper by Sohl-Dickstein, et. al, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics&...
ahxmeds's user avatar
  • 11
0 votes
0 answers
10 views

In a Diffusion Model what do I need to implement to compute the KL divergence between P and Q?

Based on the original DDPM paper, $ P_\theta(X) $ is the distribution where we draw noise from and remove it from $ X_T $ to get an image; $ Q(X) $ is the distribution where we draw noise from and add ...
David S.'s user avatar
  • 101
0 votes
0 answers
27 views

How does diffusion models "imagine" a new image from prompt?

When a new image is generated through diffusion denoising, for instance: I wonder if the resulting "Avocado Armchair" is purely using only those 2 images above it, or is drawing learned ...
James's user avatar
  • 145
0 votes
0 answers
23 views

Fine-tuned Stable Diffusion not converging with custom encoder

I'm currently fine-tuning a stable diffusion model for the task of dataset augmentation. I am training the model on 80k images from hte CelebA-attributes dataset, replacing the text encoder with ...
Tomas Premoli Muniagurria's user avatar
0 votes
0 answers
30 views

In DDPM, why is "forward posterior" needed?

From the paper: But what's the point of $q(x_t\mid x_t, x_0)$? Right before that, we already have: So we already know what $q(x_t\mid x_0)$ is for any $t$. My questions: What's the point of showing ...
xyzzyrz's user avatar
  • 101
0 votes
0 answers
36 views

How to generate an image of clothing based on the image of this clothing on some model (person) while preserving details

I have a task to generate an image of clothing based on the image of this clothing on some model (person). I tried different variations of diffusion models to reach the goal, but all of them had ...
Ararat Saribekyan's user avatar
0 votes
0 answers
15 views

When batch training a diffusion network, should we or should we not mix concepts?

Let's say we are training stable diffusion 2, tenc and unet. And we have two kinds of pictures in our dataset, apples and oranges. And we are using a batch size of 2 or higher. Should we put apples ...
Anonymous's user avatar
  • 101
0 votes
0 answers
27 views

Do I need to normalize the output latent from the autoencoder before feeding into the denoiser for latent diffusion?

I am currently learning about latent diffusion by understanding its implementation. For a standard diffusion model, the denoiser accepts a normalized image during training. During sampling at every ...
Wei Ming's user avatar
1 vote
0 answers
45 views

Why do mix models work?

Is there research on why models mixes work? One would expect that averaging the weights of two models would produce garbage, but many models mixes created by amateurs show that they not only work, but ...
allo's user avatar
  • 310
0 votes
0 answers
29 views

Are video generation also great at next frame video prediction?

If I have a good video generation model like OpenAI's new Sora, will it be capable of doing just as well at next frame video prediction?
JobHunter69's user avatar
0 votes
0 answers
46 views

What best practices for VAE do you know?

The data is binary voxel data of shape (60, 36, 60). I want to compress such data into ...
Renat Abdrakhmanov's user avatar
1 vote
0 answers
23 views

Implications of Not Zero-Centering Data in Diffusion Models: Impact on Performance and Stability?

I've been exploring the implementation and theory behind diffusion models, particularly in how they manage the latent space. A recurring theme is the emphasis on zero-centering the latent space and ...
Voyager's user avatar
  • 11
0 votes
1 answer
276 views

Why does the latent space in Stable Diffusion have a shape of 64x64x3?

Since the encoding is performed by a Variational Autoencoder, the VAE encoder must output some mean and log variance that we can ...
Renat Abdrakhmanov's user avatar
0 votes
0 answers
9 views

How to apply Latent Diffusion for 3D Binary Voxel Data?

Suppose we have a voxel of shape (60, 36, 60) with values 0 or 1 (1-occupied, 0-empty). What is the possible architecture of latent diffusion?
Renat Abdrakhmanov's user avatar
3 votes
2 answers
70 views

How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?

I've been exploring the capabilities of the Diffusion based text-to-image models and am curious about its underlying mathematical framework. Specifically, I'm interested in understanding how the model ...
hanugm's user avatar
  • 3,890
1 vote
0 answers
70 views

Are diffusion models still beneficial in highly compressed latent spaces?

Consider for example the MNIST dataset. When we apply diffusion to the pixel space, the image slowly becomes more and more noisy until white noise has been reached (like below). In the last step (t=...
Thomas Wagenaar's user avatar
1 vote
1 answer
40 views

why it is needed to generate all of the latents $z_T, z_{T-1}, z_{T-2}, ... z_1$ to finally get synthetic image x in diffusion models

As I know, we want to train a deep network to predict $\mu(z_t;w; t)$ to model $p(z_{t−1}|z_t;w)$ as $p(z_{t−1}|z_t;w) = N(z_{t−1}|\mu(z_t;w; t);\beta I)$ (here I am using notation from bishop deep ML ...
Reza_va's user avatar
  • 89
0 votes
0 answers
30 views

Where U-Net and Convolutional layers are settled in Stable Diffusion model?

When I read about Stable Diffusion model, they usually talk about adjusting convolution layers or U-Net weights. I believe they both should be related together and the U-Net is the part that accepts ...
best_of_man's user avatar
0 votes
0 answers
34 views

How to compute KL divergence in diffusion model?

In the original DDPM paper, the author suggested we can set $ \sum_{\theta} $ to a fixed value. But in the Improved DDPM paper, the author said learning the $ \sum_{\theta} $ could improve the model's ...
David S.'s user avatar
  • 101
3 votes
1 answer
67 views

How do stable diffusion models take the data into account

I'm interested in how text to image models like Midjourney and Dall-E work, where you enter a text prompt and get as output some images. I started reading some papers on it and stumbled upon "...
Rohit Pandey's user avatar
3 votes
1 answer
137 views

Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)

I am trying to understand the neural network architecture used by Ho et al. in "Denoising Diffusion Probabilistic Models" (paper, source code). They include self-attention layers in the ...
Rational Function's user avatar
3 votes
1 answer
2k views

Do LLMs based on a diffusion model (as opposed to an autoregressive model) exist?

Is there a such thing (described in title), at least in research papers (not actual models)? So far all LLMs that I know are autoregressive models.
user avatar
1 vote
0 answers
20 views

sketch guided image generation using cross-attention injection?

I have seen some papers that allow a diffusion model to generate an image that is compatible with a sketch, such as "Sketch-Guided Text-to-Image Diffusion Models" [1], which trains a module ...
Gilad Deutsch's user avatar
1 vote
1 answer
109 views

What is an information bottleneck in the context of ELBO and Hierarchical VAEs?

These slides (slide number 26) mention that the ELBO enforces an information bottleneck at the latent variables z which make it prone to bad local minima. Can you please explain what they mean by that?...
ketan dhanuka's user avatar
1 vote
2 answers
1k views

Do diffusion models take a long time to train?

I am trying to train a diffusion model (from scratch in pytorch). UNet used is not anything too fancy, takes in images and time step as input for about 512 time steps. I am using learnable embeddings ...
Aditya 's user avatar
0 votes
0 answers
63 views

DDPM - why after adding gaussian noise to image, we assume that new image is from normal distribution?

I have a question about forward process in DDPM. It is described as we sample our image from some distribution: $x_0\sim{q(x)}$ then in each time stamp $T$ we are applying gaussian noise $\epsilon\sim\...
cvzx's user avatar
  • 1
1 vote
0 answers
118 views

In the figure of Stable Diffusion, when does the switch part change?

In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
diffusion stable's user avatar
1 vote
0 answers
182 views

Diffusion Model Failing to Learn

I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
Karl's user avatar
  • 206
4 votes
5 answers
2k views

Why does Stable Diffusion use VAE instead of AE?

I am currently studying the Latent Diffusion Models (LDMs) and am interested in training my own model using a unique dataset. In my research, I came across Stable Diffusion (SD). Some sources suggest ...
P0TAT0's user avatar
  • 41
1 vote
0 answers
76 views

How does diffusion model (DDPM) ensures novel generated samples?

I am trying to understand the theoretical aspect of the denoising diffusion model. There we try to destroy the initial image x_0 through a chain of forward process and then learn a backward diffusion ...
Formal_this's user avatar
0 votes
0 answers
23 views

Training Diffusion Probabilistic Models

My question is in regards to the training process for Diffusion Probabilistic Models by Sohl-Dickenstein et. al. and also Ho et. al. and this blog post For the derivation of the model log-likelihood ...
diffusionQuestion's user avatar
1 vote
1 answer
186 views

Intuition Behind the Gradual Increase of Noise Variance in Diffusion Models

I've been studying diffusion models and came across the noise schedule, particularly how the noise variance $\beta_t$ is adjusted over iterations. I've observed that $\beta_t$ typically starts from a ...
andy90's user avatar
  • 111
0 votes
0 answers
22 views

Where to find all the documentation regarding StableDiffusion's special prompts (like mixing two persons)

I am currently playing on ClipDrop.co with Stable Diffusion, and I am confused with the tutorials I can find online. For instance, for mixing two celebrities in one pics, I read all the following ...
Myoch's user avatar
  • 101
0 votes
0 answers
58 views

How to generate quality synthetic images of human subjects to be used for training stable diffusion

I'm trying to generate some synthetic images of humans, that are "good enough" to be used to fine-tune a stable diffusion model, but i'm not sure if this is possible. I have experimented ...
interesting's user avatar
3 votes
0 answers
360 views

Relation between SDE diffusion and DDPM/DDIM

Background & Definitions In DDPM, the diffusion backward step is described as follows (where $z\sim \mathcal{N}(0,I)$ and $x_{T}\sim \mathcal{N}(0,I)$): and in DDIM we have while in the SDE ...
snatchysquid's user avatar
1 vote
1 answer
458 views

What exactly is meant by variational distribution?

What specifically does the term "variational distribution" refer to? The encoder of Variational autoencoder? Forward process of denoising diffusion probabilistic models? Images or latent ...
diffusion stable's user avatar
0 votes
1 answer
212 views

Justification of Scaling in Classifier-Free Guidance in Diffusion Models

Background | Classifier-Free Guidance Derivation To summarize the derivation of Classifier-Free Guidance, looking at this paper (Page 21.), we can write classifier guidance as: $$\nabla_{x}\log p\...
snatchysquid's user avatar
3 votes
1 answer
183 views

Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?

Consider the following diagram from the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et. al., In the context of this diagram, I'm uncertain about the ...
hanugm's user avatar
  • 3,890
0 votes
0 answers
32 views

Understanding the decreasing influence of text embedding in Text-to-Image diffusion models: A Mathematical perspective

I've been reading the paper titled eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers by Yogesh Balaji et. al. Consider the following excerpt from the abstract of the paper ...
hanugm's user avatar
  • 3,890
0 votes
0 answers
12 views

Clarification regarding the training status of 'domain specific encoder' in stable diffusion

I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms. In ...
hanugm's user avatar
  • 3,890
1 vote
1 answer
51 views

In-depth understanding of formulation and guidance mechanisms in Diffusion models

I've been reading a research paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. and came across an a concept related to diffusion models (DMs). In the ...
hanugm's user avatar
  • 3,890
1 vote
1 answer
196 views

Math behind Diffusion models explanation?

I am recently reading this wonderful article https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#what-are-diffusion-models about the math behind the diffusion models , As i dont have a ...
mat's user avatar
  • 13
0 votes
2 answers
123 views

How to expand reconstruction error to mean squared error in Variational AutoEncoder? [closed]

How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$? [reconstruction error] $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$...
diffusion stable's user avatar