Questions tagged [diffusion-models]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

DDPM - why after adding gaussian noise to image, we assume that new image is from normal distribution?

I have a question about forward process in DDPM. It is described as we sample our image from some distribution: $x_0\sim{q(x)}$ then in each time stamp $T$ we are applying gaussian noise $\epsilon\sim\...
cvzx's user avatar
  • 1
-1 votes
0 answers
28 views

Diffusion model for image to image translation

I am interested in using the diffusion model for image-to-image translation (pair images). I used this repository for semantic image synthesis via diffusion models Code. How can I use it for image-to-...
yun dan's user avatar
1 vote
0 answers
60 views

In the figure of Stable Diffusion, when does the switch part change?

In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
diffusion stable's user avatar
1 vote
0 answers
76 views

Diffusion Model Failing to Learn

I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
Karl's user avatar
  • 206
1 vote
2 answers
143 views

Why does Stable Diffusion use VAE instead of AE?

I am currently studying the Latent Diffusion Models (LDMs) and am interested in training my own model using a unique dataset. In my research, I came across Stable Diffusion (SD). Some sources suggest ...
P0TAT0's user avatar
  • 11
1 vote
0 answers
36 views

How does diffusion model (DDPM) ensures novel generated samples?

I am trying to understand the theoretical aspect of the denoising diffusion model. There we try to destroy the initial image x_0 through a chain of forward process and then learn a backward diffusion ...
Formal_this's user avatar
0 votes
0 answers
20 views

Training Diffusion Probabilistic Models

My question is in regards to the training process for Diffusion Probabilistic Models by Sohl-Dickenstein et. al. and also Ho et. al. and this blog post For the derivation of the model log-likelihood ...
diffusionQuestion's user avatar
1 vote
1 answer
44 views

Intuition Behind the Gradual Increase of Noise Variance in Diffusion Models

I've been studying diffusion models and came across the noise schedule, particularly how the noise variance $\beta_t$ is adjusted over iterations. I've observed that $\beta_t$ typically starts from a ...
andy90's user avatar
  • 111
0 votes
0 answers
19 views

Where to find all the documentation regarding StableDiffusion's special prompts (like mixing two persons)

I am currently playing on ClipDrop.co with Stable Diffusion, and I am confused with the tutorials I can find online. For instance, for mixing two celebrities in one pics, I read all the following ...
Myoch's user avatar
  • 101
0 votes
0 answers
33 views

How to generate quality synthetic images of human subjects to be used for training stable diffusion

I'm trying to generate some synthetic images of humans, that are "good enough" to be used to fine-tune a stable diffusion model, but i'm not sure if this is possible. I have experimented ...
interesting's user avatar
3 votes
0 answers
138 views

Relation between SDE diffusion and DDPM/DDIM

Background & Definitions In DDPM, the diffusion backward step is described as follows (where $z\sim \mathcal{N}(0,I)$ and $x_{T}\sim \mathcal{N}(0,I)$): and in DDIM we have while in the SDE ...
snatchysquid's user avatar
0 votes
1 answer
97 views

What exactly is meant by variational distribution?

What specifically does the term "variational distribution" refer to? The encoder of Variational autoencoder? Forward process of denoising diffusion probabilistic models? Images or latent ...
diffusion stable's user avatar
0 votes
1 answer
120 views

Justification of Scaling in Classifier-Free Guidance in Diffusion Models

Background | Classifier-Free Guidance Derivation To summarize the derivation of Classifier-Free Guidance, looking at this paper (Page 21.), we can write classifier guidance as: $$\nabla_{x}\log p\...
snatchysquid's user avatar
0 votes
0 answers
49 views

Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?

Consider the following diagram from the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et. al., In the context of this diagram, I'm uncertain about the ...
hanugm's user avatar
  • 3,612
0 votes
0 answers
24 views

Understanding the decreasing influence of text embedding in Text-to-Image diffusion models: A Mathematical perspective

I've been reading the paper titled eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers by Yogesh Balaji et. al. Consider the following excerpt from the abstract of the paper ...
hanugm's user avatar
  • 3,612
0 votes
0 answers
8 views

Clarification regarding the training status of 'domain specific encoder' in stable diffusion

I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms. In ...
hanugm's user avatar
  • 3,612
1 vote
1 answer
38 views

In-depth understanding of formulation and guidance mechanisms in Diffusion models

I've been reading a research paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. and came across an a concept related to diffusion models (DMs). In the ...
hanugm's user avatar
  • 3,612
0 votes
1 answer
95 views

Math behind Diffusion models explanation?

I am recently reading this wonderful article https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#what-are-diffusion-models about the math behind the diffusion models , As i dont have a ...
mat's user avatar
  • 3
0 votes
2 answers
93 views

How to expand reconstruction error to mean squared error in Variational AutoEncoder? [closed]

How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$? [reconstruction error] $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$...
diffusion stable's user avatar
0 votes
0 answers
104 views

Stable Diffusion: Is it possible to merge specific token weights from one model into another?

I have three models: A, B, C I merged A+C and B+C together. A+C gives me great results for certain things but is not so good with specific other things. B+C is great with things that A+C is not, but ...
SquareCat's user avatar
  • 101
1 vote
1 answer
72 views

Modern graduate-level machine learning books with focus on generative models

I'm looking for a modern machine learning book with graduate-level treatment of more recent topics such as diffusion and generative models, transformers etc. I have a hard copy of Deep Learning by ...
user74376's user avatar
  • 111
0 votes
0 answers
35 views

In DDPM model, why authors subtract the largest value along j dimension of q*k values matrices?

I have a question about a part of DDPM model code. Below is the Attention class of DDPM model from this website. My questions is at the bottom. ...
Jun's user avatar
  • 3
1 vote
0 answers
58 views

Samples from a reverse diffusion process with cosine noise schedule blow up

I have implemented a diffusion probabilistic model, and I am finding some of the model behavior unexpected. When I draw samples from an untrained reverse diffusion process with 20 denoising steps ...
Jabby's user avatar
  • 11
0 votes
0 answers
54 views

How to use diffusion models to draw on top of image?

I am thinking about a pipeline that involves multiple diffusion networks. The first one generates an image from a prompt, a use case that is well-documented in literature. The following networks ...
emilaz's user avatar
  • 133
3 votes
1 answer
399 views

What's the architecture that allows the generation of new images based on input image in tools like Midjourney?

I understand that the high-level architecture of tools like Midjourney use diffusion models to generate images from text. What I don't understand is which type of network architecture allow the second ...
emilaz's user avatar
  • 133
0 votes
0 answers
38 views

Should the norm of samples from an untrained denoising diffusion probabilistic model scale with number of diffusion steps?

I have implemented a diffusion probabilistic model for the first time, and I am finding some of the model behavior undesirable, but I am not sure if it is normal or the result of an incorrect ...
Jabby's user avatar
  • 11
2 votes
2 answers
2k views

About cosine noise schedule in Diffusion Model

Could you provide the explanation of Figure 4 from the paper at Improved Denoising Diffusion Probabilistic Models? (1) The paper says, "the end of the forward noising process is too noisy,and so ...
diffusion stable's user avatar
1 vote
1 answer
252 views

How to train Diffusion model with additional loss?

I would like to train a diffusion model with an additional loss on the created image. Without getting into too much details my intention is to do something like regularization, for example you may ...
Gilad Deutsch's user avatar
0 votes
1 answer
345 views

In the original diffusion model paper, why do they sample the first step with the same loss?

In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a ...
Grieverheart's user avatar
0 votes
0 answers
38 views

DDIM scheduler scaling the denoising model input depending on the current timestep

I can't find any resources on scaling the denoising model input depending on the current timestep for diffusion models. Just briefly mentioned in the appendix of the paper , but not much more. Does ...
SumakuTension's user avatar
0 votes
1 answer
191 views

In DDPM, why can't we get $x_0$ from $x_t$ directly by using equation?

In DDPM, when given $x_{t}$ and time step $t$, U-Net predicts $\epsilon$, if I understand correctly, this $\epsilon$ is the exact same $\epsilon$ in equation $$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \...
neese's user avatar
  • 1
0 votes
0 answers
44 views

Layer Questions regarding Bidirectional VAE (D3VAE)

I am currently trying to figure out how D3VAE are working, but I can't seem to understand the network architecture given. The paper can be found here: https://openreview.net/pdf?id=rG0jm74xtx The ...
Patrick Lehnen's user avatar
0 votes
0 answers
73 views

Why learning with backward process is need in diffusion model despite knowing $q(x_{t-1}|x_t)$

I recently learnt about the diffusion model in deep learning, can someone explain to me if we can induce noise to an input data and make it a gaussian like data, why can't we use the same process in ...
aaa's user avatar
  • 1
0 votes
0 answers
123 views

Diffusion Models - Loss normalization for different noise schedules

With reference to the paper "Denoising Diffusion Probabilistic Models", and running the code given in this notebook showing a simple example on "flattened" data, I have the ...
Peblo's user avatar
  • 31
0 votes
0 answers
123 views

Resources to understand the math behind diffusion models

I started learning about diffusion models, but I couldn't follow the math proofs (probability distribution terms and other stuff), is there any sources you suggest to understand what they're talking ...
hakim47's user avatar
2 votes
1 answer
52 views

Clarification on the training objective of denoising diffusion models

I'm reading the Denoising Diffusion Probabilistic Models paper (Ho et al. 2020). And I am puzzled about the training objective. I understood (I think) the trick regarding the reparametrization of the ...
user3903647's user avatar
0 votes
0 answers
62 views

what's the benefit of diffusion models reverse process

To draw a picture, we have two approaches: gradually add(and modify) pixels on white paper. gradually remove(and modify) pixels from a complete random noise picture. Apparently, the diffusion model ...
zzzgoo's user avatar
  • 151
0 votes
1 answer
100 views

Blurring of image in generative model using diffusion probabilistic method

In the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" by Sohl-Dickstein et al. on diffusion for generative models, Figure 1 shows the forward trajectory for a 2-...
sunfishstanford's user avatar
0 votes
0 answers
19 views

What does it take to create quality website illustrations using AI?

I would like to create illustrations quickly using text prompts. I have tried using DALL-E, which follows the prompt well enough, but the quality of the image details is not good. I have also tried ...
Olek Gornostal's user avatar
3 votes
0 answers
194 views

Reverse Process in Diffusion Model Doesn't Return Original Image

I am attempting to program a Denoising Diffusion Model based on the one introduced in the article by Ho et al. (2020). However, I have run into issues while testing the reverse diffusion process. ...
cabralpinto's user avatar
1 vote
0 answers
391 views

Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset

I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they ...
sb3's user avatar
  • 137
0 votes
2 answers
86 views

How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?

Regarding Eq. (1) in Progressive Distillation for Fast Sampling of Diffusion Models, $$ q(\mathbf{z}_t|\mathbf{z}_s) = \mathcal{N}(\mathbf{z}_t; (\alpha_t/\alpha_s)\mathbf{z}_s, \sigma _{t|s}^2 \...
hotohoto's user avatar
  • 121
1 vote
0 answers
979 views

How to use diffusers StableDiffusionImg2ImgPipeline with "Inpainting conditioning mask strength 0-1" and an inpainting.ckpt model?

I'm working on a project & want to change the art style of images but keep them close to their original form. As shown in the image, this can be done using https://github.com/AUTOMATIC1111/stable-...
Alexandru Pietroiu's user avatar
0 votes
0 answers
580 views

Do tensor cores provide advantages for running Stable Diffusion or only for training?

If I am only interested in running Stable Diffusion, using pre-trained weights, to generate images, are there any advantages to using a GPU with more Tensor cores? Or will any CUDA-compatible GPU ...
sak's user avatar
  • 101
1 vote
2 answers
111 views

Reverse Distribution in Denoising Diffusion Models is Simple

In explanations of denoising diffusion models it is stated that $q(x_{t-1}|x_t)$ is intractable. This is often justified via Bayes' rule, i.e. $$ q(x_{t-1}|x_t) \propto q(x_t|x_{t-1})q(x_{t-1}) $$ and ...
Patrick Johnstone's user avatar
1 vote
0 answers
136 views

Is it possible to combine DDPM with GAN?

From what I understand in GAN, the main idea is that you have a generator and a discriminator network that are "competing" with each other. The generator trying to make images that the ...
Nikita Belooussov's user avatar
-1 votes
1 answer
499 views

Is Diffusion model instable during the training? [closed]

I ran an experiment using a diffusion model (SR3 model) and got good results (experiment using jupyter notebook). Then, after running the same notebook many times with the same configuration, I did ...
ProEns08's user avatar
  • 101
2 votes
3 answers
398 views

Forward Diffusion Process Derivation In Diffusion Models

In papers and other material regarding diffusion models the forward diffusion process is defined by adding a small amount of Gaussian noise to an image $x_0$ for $T$ time steps. In each time step the ...
nkam's user avatar
  • 21
1 vote
1 answer
87 views

How is the variance for a diffusion kernel derived for a diffusion model?

So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward ...
Cynthia Kim's user avatar
10 votes
2 answers
4k views

Using AI to extend an imagine pattern

I have created some nice patterns using the MidJourney tool. I'd like to find a way to extend these patterns, and I was thinking about an AI tool that takes one of these patterns and extends it in all ...
Nicola Lepetit's user avatar