15 votes
Accepted

Using AI to extend an imagine pattern

As Edoardo says in their excellent answer, the task at hand can be approached as an outpainting problem and there's some great tools available to do this. To throw an alternative into the ring, I'd ...
James Ashford's user avatar
9 votes

Using AI to extend an imagine pattern

The task you would like to accomplish is referred to as "outpainting". See example below. Very recently, OpenAI released an outpainting feature that extends the possible operations to ...
Edoardo Guerriero's user avatar
7 votes
Accepted

Do LLMs based on a diffusion model (as opposed to an autoregressive model) exist?

This is a false dichotomy. Most diffusion models, including Dalle 2 and 3, already are transformers. However, assuming you meant to ask if any language models use diffusion as opposed to a GPT, the ...
chessprogrammer's user avatar
4 votes
Accepted

What's the architecture that allows the generation of new images based on input image in tools like Midjourney?

Midjourney (and DALL-E 2 I think) uses a concept vector (or "embedding") to condition its image outputs, which can be produced in at least two ways: By summarising text input By converting ...
Neil Slater's user avatar
  • 31.5k
3 votes

How to expand reconstruction error to mean squared error in Variational AutoEncoder?

In a way, you're right. The reconstruction loss is just an idea because you have not yet defined the distribution $p_\theta$. If you assume that this distribution is e.g. a Gaussian, then you should ...
nbro's user avatar
  • 40.2k
3 votes
Accepted

Is Diffusion model instable during the training?

This is (mostly) because of random weight initialization, each time you instance your model, starting weights are different, and during training the model weights converge to a different local minima. ...
Dr. Snoopy's user avatar
  • 1,320
2 votes
Accepted

About cosine noise schedule in Diffusion Model

I will suppose that you already have understood how diffusion models work. Some good resources are this blog and the DDPM paper. If we look at Figure 3 of the paper, we see that in linear schedule the ...
Ciodar's user avatar
  • 390
2 votes

What exactly is meant by variational distribution?

The variational distribution is the distribution (or set of distributions) that you use to approximate the distribution you are looking for. It's often denoted by $q$, $q_\phi$ or $q_\phi(z \mid x)$, ...
nbro's user avatar
  • 40.2k
2 votes

In-depth understanding of formulation and guidance mechanisms in Diffusion models

You can think as if the network learns the gradient of the data distribution... For example, think about having some points in 1D which are distributed as 2 Gaussians: Learning the gradient means ...
Alberto's user avatar
  • 1,677
2 votes

why it is needed to generate all of the latents $z_T, z_{T-1}, z_{T-2}, ... z_1$ to finally get synthetic image x in diffusion models

The idea behind this approach is a bit like trying to untangle a complex knot step by step, rather than trying to untangle it all at once. When we add noise to an original image to create a "...
Victor Roza's user avatar
2 votes

How do stable diffusion models take the data into account

I have recently taken a seminar on a similar let me explain you in brief, The complete end-to-end process has 3 steps while inferencing: Text Encoding (using CLIP Model) Image Information Creator (...
Hiren Namera's user avatar
2 votes

How to expand reconstruction error to mean squared error in Variational AutoEncoder?

If you model the distribution $p_\theta$ as Gaussian then: $\displaystyle p_\theta(x|z) = \frac{1}{\sigma\sqrt{2\pi}}e^{-\frac{1}{2}\frac{x-\mu}{\sigma}^2} $ $\displaystyle \log p_\theta(x|z) = -log \...
pi-tau's user avatar
  • 692
2 votes
Accepted

Why does the latent space in Stable Diffusion have a shape of 64x64x3?

There's not really a restriction on the shape for variational autoencoders. If you really wanted a 1D vector, you could just flatten the matrix and get a vector of size ...
Alexander Wan's user avatar
1 vote

How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?

I'm only familiar with Stable Diffusion 1.x models, but I assume that all LDMs work fundamentally the same way. In my understanding the mapping from input text to output image is a deterministic ...
NikoNyrh's user avatar
  • 757
1 vote

How does diffusion based text-to-image generation models Mathematically classify inputs to outputs?

The paper you are referencing actually already achieving text-to-image generation. It's a tricky question in my opinion, since the process that is modelled by neural network during denoising process ...
vl_knd's user avatar
  • 474
1 vote
Accepted

Understanding the function of attention layers in a convolutional neural network (U-Net in a diffusion model)

The implementation of self-attention in the source code for the "Self-Attention Generative Adversarial Networks" (SAGAN) paper is somewhat easier to follow than that in the "Denoising ...
Rational Function's user avatar
1 vote

Math behind Diffusion models explanation?

That exponential comes from the PDF of the assumed distribution that you might have missed in the blogpost: and plug those PDF in that bayes formula and you will end up with those equations
Alberto's user avatar
  • 1,677
1 vote

What is an information bottleneck in the context of ELBO and Hierarchical VAEs?

In the slides, they aim to explain the use of Hierarchical VAEs and the types of problems they address. "Evidence Lower Bound" (ELBO) measures a lower bound approximation of the log-...
Cesar Ruiz's user avatar
1 vote

Do diffusion models take a long time to train?

Diffusion models are very data hungry. Without data augmentation that won't be enough to train it nor would I expect realistic images from a dataset that small for something so complicated. Diffusion ...
Kode's user avatar
  • 111
1 vote

Why does Stable Diffusion use VAE instead of AE?

Yes, SD uses VAEs instead of AEs in learning process. There are reasons behind it like AEs create deterministic latent space representation. That means for any input given AEs produce only specific ...
Hiren Namera's user avatar
1 vote

Why does Stable Diffusion use VAE instead of AE?

Stabilizing diffusion uses variational autoencoders (VAEs) instead of autoencoders (AEs) because VAEs allow for the generation of continuously distributed representations in the latent space, which ...
kathy's user avatar
  • 11
1 vote

Why does Stable Diffusion use VAE instead of AE?

To my knowledge, when it comes to stable diffusion, the VQ-VAE is the commonly used method. This differs slightly from vanilla VAE which assumes the encoded features to be a normal distribution and ...
Jack123's user avatar
  • 11
1 vote

Modern graduate-level machine learning books with focus on generative models

Berkeley CS294-158 is a graduate-level course on deep unsupervised learning. They cover a lot of architectures used in modern generative modeling. They have recorded lectures and slides online. ...
Alexander Wan's user avatar
1 vote

About cosine noise schedule in Diffusion Model

Intuitively, the noise schedule shall be smooth and progressive enough to be easily approximated by the NN. In this context, a linear schedule is plausibly too "easy", and hence prone to ...
Peblo's user avatar
  • 31
1 vote

How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?

There is another derivation: $$ {\begin{aligned} \sigma_{t|s}^2 &:= \sigma_t^2 - {\alpha_{t}^2 \over \alpha_s^2} \sigma_s^2 \\ &= \left( 1 - {\alpha_t^2 \sigma_s^2\over \sigma_t^2\alpha_s^2} \...
hotohoto's user avatar
  • 121
1 vote

How to derive the variance of the forward step of Variational Diffusion Models in terms of the log signal-to-noise ratio $\lambda_t$?

One of my colleagues gave me a solution. (Thanks Ken!) $$ {\begin{aligned} \sigma_{t|s}^2 &= 1 - \alpha_{t|s}^2 \\ &= 1 - {\alpha_t^2 \over \alpha_s^2} \\ &= {\alpha_s^2 - \alpha_t^2 \over ...
hotohoto's user avatar
  • 121
1 vote

Reverse Distribution in Denoising Diffusion Models is Simple

while we set $R$ to be independent from $x_{t-1}$ in the calculation of $x_t$, we no long have independency between $x_t$ and $R$, and I guess this gives rise to the confusion. For simplicity, let us ...
Yi Huang's user avatar
  • 111
1 vote
Accepted

Forward Diffusion Process Derivation In Diffusion Models

The relationship between $x_t$ and $x_{t-1}$ is as follows: $$ x_t = \sqrt{1-\beta_t}x_{t-1}+\sqrt{\beta_t}\epsilon_t,\quad \epsilon_t\sim\mathcal{N}(0,I). $$ Not only is a small amount of noise added,...
Patrick Johnstone's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible