Questions tagged [diffusion-models]
The diffusion-models tag has no usage guidance.
50
questions
0
votes
0
answers
15
views
DDPM - why after adding gaussian noise to image, we assume that new image is from normal distribution?
I have a question about forward process in DDPM. It is described as we sample our image from some distribution: $x_0\sim{q(x)}$ then in each time stamp $T$ we are applying gaussian noise $\epsilon\sim\...
- 1
-1
votes
0
answers
28
views
Diffusion model for image to image translation
I am interested in using the diffusion model for image-to-image translation (pair images). I used this repository for semantic image synthesis via diffusion models Code. How can I use it for image-to-...
- 1
1
vote
0
answers
60
views
In the figure of Stable Diffusion, when does the switch part change?
In the illustration of Stable Diffusion, there is a concatenation part through Cross-Attention. Why is there a switch in the concatenation part?
1
vote
0
answers
76
views
Diffusion Model Failing to Learn
I'm trying to train a diffusion model to map between paired embedding spaces - ie using a CLIP text embedding to predict a CLIP image embedding. I have a working baseline model that predicts the ...
- 206
1
vote
2
answers
143
views
Why does Stable Diffusion use VAE instead of AE?
I am currently studying the Latent Diffusion Models (LDMs) and am interested in training my own model using a unique dataset. In my research, I came across Stable Diffusion (SD). Some sources suggest ...
- 11
1
vote
0
answers
36
views
How does diffusion model (DDPM) ensures novel generated samples?
I am trying to understand the theoretical aspect of the denoising diffusion model. There we try to destroy the initial image x_0 through a chain of forward process and then learn a backward diffusion ...
- 135
0
votes
0
answers
20
views
Training Diffusion Probabilistic Models
My question is in regards to the training process for Diffusion Probabilistic Models by Sohl-Dickenstein et. al. and also Ho et. al. and this blog post
For the derivation of the model log-likelihood ...
1
vote
1
answer
44
views
Intuition Behind the Gradual Increase of Noise Variance in Diffusion Models
I've been studying diffusion models and came across the noise schedule, particularly how the noise variance $\beta_t$ is adjusted over iterations. I've observed that $\beta_t$ typically starts from a ...
- 111
0
votes
0
answers
19
views
Where to find all the documentation regarding StableDiffusion's special prompts (like mixing two persons)
I am currently playing on ClipDrop.co with Stable Diffusion, and I am confused with the tutorials I can find online.
For instance, for mixing two celebrities in one pics, I read all the following ...
- 101
0
votes
0
answers
33
views
How to generate quality synthetic images of human subjects to be used for training stable diffusion
I'm trying to generate some synthetic images of humans, that are "good enough" to be used to fine-tune a stable diffusion model, but i'm not sure if this is possible.
I have experimented ...
3
votes
0
answers
138
views
Relation between SDE diffusion and DDPM/DDIM
Background & Definitions
In DDPM, the diffusion backward step is described as follows (where $z\sim \mathcal{N}(0,I)$ and $x_{T}\sim \mathcal{N}(0,I)$):
and in DDIM we have
while in the SDE ...
- 79
0
votes
1
answer
97
views
What exactly is meant by variational distribution?
What specifically does the term "variational distribution" refer to?
The encoder of Variational autoencoder?
Forward process of denoising diffusion probabilistic models?
Images or latent ...
0
votes
1
answer
120
views
Justification of Scaling in Classifier-Free Guidance in Diffusion Models
Background | Classifier-Free Guidance Derivation
To summarize the derivation of Classifier-Free Guidance, looking at this paper (Page 21.), we can write classifier guidance as: $$\nabla_{x}\log p\...
- 79
0
votes
0
answers
49
views
Understanding the functionality of the switch in the latent diffusion models: Does conditioning information pass to both cross attention and $z_{T}$?
Consider the following diagram from the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et. al.,
In the context of this diagram, I'm uncertain about the ...
- 3,612
0
votes
0
answers
24
views
Understanding the decreasing influence of text embedding in Text-to-Image diffusion models: A Mathematical perspective
I've been reading the paper titled eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers by Yogesh Balaji et. al. Consider the following excerpt from the abstract of the paper
...
- 3,612
0
votes
0
answers
8
views
Clarification regarding the training status of 'domain specific encoder' in stable diffusion
I am currently studying the paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. Specifically, I am focused on the section 3.3. Conditioning Mechanisms.
In ...
- 3,612
1
vote
1
answer
38
views
In-depth understanding of formulation and guidance mechanisms in Diffusion models
I've been reading a research paper titled High-Resolution Image Synthesis with Latent Diffusion Models by Robin Rombach et al. and came across an a concept related to diffusion models (DMs). In the ...
- 3,612
0
votes
1
answer
95
views
Math behind Diffusion models explanation?
I am recently reading this wonderful article https://lilianweng.github.io/posts/2021-07-11-diffusion-models/#what-are-diffusion-models about the math behind the diffusion models ,
As i dont have a ...
- 3
0
votes
2
answers
93
views
How to expand reconstruction error to mean squared error in Variational AutoEncoder? [closed]
How to expand reconstruction error to mean squared error when it is $\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$?
[reconstruction error]
$\mathbb{E}_{z\sim q_{\phi}(z|x)}[\log p_\theta(x|z)]$...
0
votes
0
answers
104
views
Stable Diffusion: Is it possible to merge specific token weights from one model into another?
I have three models: A, B, C
I merged A+C and B+C together.
A+C gives me great results for certain things but is not so good with specific other things. B+C is great with things that A+C is not, but ...
- 101
1
vote
1
answer
72
views
Modern graduate-level machine learning books with focus on generative models
I'm looking for a modern machine learning book with graduate-level treatment of more recent topics such as diffusion and generative models, transformers etc.
I have a hard copy of Deep Learning by ...
- 111
0
votes
0
answers
35
views
In DDPM model, why authors subtract the largest value along j dimension of q*k values matrices?
I have a question about a part of DDPM model code.
Below is the Attention class of DDPM model from this website. My questions is at the bottom.
...
- 3
1
vote
0
answers
58
views
Samples from a reverse diffusion process with cosine noise schedule blow up
I have implemented a diffusion probabilistic model, and I am finding some of the model behavior unexpected.
When I draw samples from an untrained reverse diffusion process with 20 denoising steps ...
- 11
0
votes
0
answers
54
views
How to use diffusion models to draw on top of image?
I am thinking about a pipeline that involves multiple diffusion networks. The first one generates an image from a prompt, a use case that is well-documented in literature. The following networks ...
- 133
3
votes
1
answer
399
views
What's the architecture that allows the generation of new images based on input image in tools like Midjourney?
I understand that the high-level architecture of tools like Midjourney use diffusion models to generate images from text. What I don't understand is which type of network architecture allow the second ...
- 133
0
votes
0
answers
38
views
Should the norm of samples from an untrained denoising diffusion probabilistic model scale with number of diffusion steps?
I have implemented a diffusion probabilistic model for the first time, and I am finding some of the model behavior undesirable, but I am not sure if it is normal or the result of an incorrect ...
- 11
2
votes
2
answers
2k
views
About cosine noise schedule in Diffusion Model
Could you provide the explanation of Figure 4 from the paper at Improved Denoising Diffusion Probabilistic Models?
(1) The paper says, "the end of the forward noising process is too noisy,and so ...
1
vote
1
answer
252
views
How to train Diffusion model with additional loss?
I would like to train a diffusion model with an additional loss on the created image. Without getting into too much details my intention is to do something like regularization, for example you may ...
- 639
0
votes
1
answer
345
views
In the original diffusion model paper, why do they sample the first step with the same loss?
In the original diffusion model paper by Sohl-Dickstein et al., they explain very little about calculating the loss and training and network to learn the diffusion process. They did publish a ...
- 101
0
votes
0
answers
38
views
DDIM scheduler scaling the denoising model input depending on the current timestep
I can't find any resources on scaling the denoising model input depending on the current timestep for diffusion models. Just briefly mentioned in the appendix of the paper , but not much more. Does ...
- 265
0
votes
1
answer
191
views
In DDPM, why can't we get $x_0$ from $x_t$ directly by using equation?
In DDPM, when given $x_{t}$ and time step $t$, U-Net predicts $\epsilon$, if I understand correctly, this $\epsilon$ is the exact same $\epsilon$ in equation
$$x_{t}=\sqrt{\bar\alpha_{t}}*x_{0} + \...
- 1
0
votes
0
answers
44
views
Layer Questions regarding Bidirectional VAE (D3VAE)
I am currently trying to figure out how D3VAE are working, but I can't seem to understand the network architecture given. The paper can be found here:
https://openreview.net/pdf?id=rG0jm74xtx
The ...
0
votes
0
answers
73
views
Why learning with backward process is need in diffusion model despite knowing $q(x_{t-1}|x_t)$
I recently learnt about the diffusion model in deep learning, can someone explain to me if we can induce noise to an input data and make it a gaussian like data, why can't we use the same process in ...
- 1
0
votes
0
answers
123
views
Diffusion Models - Loss normalization for different noise schedules
With reference to the paper "Denoising Diffusion Probabilistic Models", and running the code given in this notebook showing a simple example on "flattened" data, I have the ...
- 31
0
votes
0
answers
123
views
Resources to understand the math behind diffusion models
I started learning about diffusion models, but I couldn't follow the math proofs (probability distribution terms and other stuff), is there any sources you suggest to understand what they're talking ...
- 1
2
votes
1
answer
52
views
Clarification on the training objective of denoising diffusion models
I'm reading the Denoising Diffusion Probabilistic Models paper (Ho et al. 2020). And I am puzzled about the training objective. I understood (I think) the trick regarding the reparametrization of the ...
- 21
0
votes
0
answers
62
views
what's the benefit of diffusion models reverse process
To draw a picture, we have two approaches:
gradually add(and modify) pixels on white paper.
gradually remove(and modify) pixels from a complete random noise picture.
Apparently, the diffusion model ...
- 151
0
votes
1
answer
100
views
Blurring of image in generative model using diffusion probabilistic method
In the 2015 paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" by Sohl-Dickstein et al. on diffusion for generative models, Figure 1 shows the forward trajectory for a 2-...
0
votes
0
answers
19
views
What does it take to create quality website illustrations using AI?
I would like to create illustrations quickly using text prompts. I have tried using DALL-E, which follows the prompt well enough, but the quality of the image details is not good. I have also tried ...
- 101
3
votes
0
answers
194
views
Reverse Process in Diffusion Model Doesn't Return Original Image
I am attempting to program a Denoising Diffusion Model based on the one introduced in the article by Ho et al. (2020). However, I have run into issues while testing the reverse diffusion process.
...
- 31
1
vote
0
answers
391
views
Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset
I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they ...
- 137
0
votes
2
answers
86
views
How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?
Regarding Eq. (1) in Progressive Distillation for Fast Sampling of Diffusion Models,
$$
q(\mathbf{z}_t|\mathbf{z}_s) = \mathcal{N}(\mathbf{z}_t; (\alpha_t/\alpha_s)\mathbf{z}_s, \sigma _{t|s}^2 \...
- 121
1
vote
0
answers
979
views
How to use diffusers StableDiffusionImg2ImgPipeline with "Inpainting conditioning mask strength 0-1" and an inpainting.ckpt model?
I'm working on a project & want to change the art style of images but keep them close to their original form.
As shown in the image, this can be done using https://github.com/AUTOMATIC1111/stable-...
0
votes
0
answers
580
views
Do tensor cores provide advantages for running Stable Diffusion or only for training?
If I am only interested in running Stable Diffusion, using pre-trained weights, to generate images, are there any advantages to using a GPU with more Tensor cores? Or will any CUDA-compatible GPU ...
- 101
1
vote
2
answers
111
views
Reverse Distribution in Denoising Diffusion Models is Simple
In explanations of denoising diffusion models it is stated that $q(x_{t-1}|x_t)$ is intractable. This is often justified via Bayes' rule, i.e.
$$
q(x_{t-1}|x_t) \propto q(x_t|x_{t-1})q(x_{t-1})
$$
and ...
1
vote
0
answers
136
views
Is it possible to combine DDPM with GAN?
From what I understand in GAN, the main idea is that you have a generator and a discriminator network that are "competing" with each other. The generator trying to make images that the ...
-1
votes
1
answer
499
views
Is Diffusion model instable during the training? [closed]
I ran an experiment using a diffusion model (SR3 model) and got good results (experiment using jupyter notebook). Then, after running the same notebook many times with the same configuration, I did ...
- 101
2
votes
3
answers
398
views
Forward Diffusion Process Derivation In Diffusion Models
In papers and other material regarding diffusion models the forward diffusion process is defined by adding a small amount of Gaussian noise to an image $x_0$ for $T$ time steps. In each time step the ...
- 21
1
vote
1
answer
87
views
How is the variance for a diffusion kernel derived for a diffusion model?
So I'm watching this video tutorial from CVPR this year on diffusion models, and I am confused by the variance term in the distribution on the left on the video. I understand that in the forward ...
- 11
10
votes
2
answers
4k
views
Using AI to extend an imagine pattern
I have created some nice patterns using the MidJourney tool. I'd like to find a way to extend these patterns, and I was thinking about an AI tool that takes one of these patterns and extends it in all ...
- 203