15
votes
Why are LLMs able to reproduce bodies of known text exactly?
Google "call me ishmael. some years ago—never mind how long precisely—having" and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, ...
- 3,112
13
votes
Why are LLMs able to reproduce bodies of known text exactly?
LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the "decompression" of parts to be recognizable and reasonably faithful. I ...
12
votes
Accepted
Is plain autoencoder a generative model?
An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the ...
- 1,748
10
votes
Accepted
What is the meaning of $V(D,G)$ in the GAN objective function?
To understand this equation first you need to understand the context in which it is first introduced. We have two neural networks (i.e. $D$ and $G$) that are playing a minimax game. This means that ...
- 3,203
7
votes
Accepted
How can we process the data from both the true distribution and the generator?
The Focus of This Question
"How can ... we process the data from the true distribution and the data from the generative model in the same iteration?
Analyzing the Foundational Publication
In the ...
- 7,513
6
votes
Why diffusion model always use U-Net?
I don't have a definitive answer but I'd state my intuitions anyways:
Diffusion models are highly related to the idea of stacked denoising autoencoders [Kumar et al. (2014)]. Additionally, U-Net-like ...
- 1,748
5
votes
Are deep learning models suitable for training with sparse data?
The problem isn't the GAN but the implementation of its discriminator which is typically a convolutional neural network (CNN). CNNs have trouble with sparse data. They require dense data to learn ...
- 1,937
4
votes
Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?
The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
- 49
4
votes
Accepted
Other deep learning image generation techniques besides GANs?
There are several generative models that have been proposed before or roughly at the same time of the GAN (2014). For example, the deep Boltzman machine (2009), deep generative stochastic network (...
- 41.1k
4
votes
Accepted
How can we find find the input image which maximizes the class-probability for an ANN?
In deep networks there is actually a wide variety of solutions to the problem, but if you need to find one, any easy way to do this is just through normal optimization schemes
$$\hat x = argmin_x \ L(...
- 2,379
4
votes
How can we find find the input image which maximizes the class-probability for an ANN?
Probably the simplest way to search for an image with the highest probability of being a cat is to use a technique similar to Deep Dream:
Load the network for training, but freeze all the network ...
- 32.9k
4
votes
Accepted
Where is the mistake in my derivation of the GAN loss function?
I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
- 809
4
votes
Accepted
Would it be possible to determine the dataset a neural network was trained on?
You can already do this with some neural networks, such as GANs and VAEs, which are generative models that learn a probability distribution over the inputs, so they learn how to produce e.g. images ...
- 41.1k
4
votes
Book(s) on generative models
From the theoretical foundations one can look into the Chapter 20: Deep Generative Models of the classic DL book by Goodfellow, Bengio https://amzn.to/2MmZNbH. Not ...
- 2,724
4
votes
Accepted
How to determine the quality of synthetic data?
Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
- 258
4
votes
Accepted
Why don't we also need to approximate $p(x \mid z)$ in the VAE?
What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right?
You could assume $p(x)$ is ...
- 41.1k
4
votes
What makes ChatGPT a generative model?
What are generative (and discriminative) models?
If the model learns a distribution of the form $p(x)$ or $p(x, y)$, where $x$ are the inputs and $y$ the outputs/labels, from which you can sample data,...
- 41.1k
4
votes
Accepted
What's the architecture that allows the generation of new images based on input image in tools like Midjourney?
Midjourney (and DALL-E 2 I think) uses a concept vector (or "embedding") to condition its image outputs, which can be produced in at least two ways:
By summarising text input
By converting ...
- 32.9k
4
votes
Is there a limitation to the amount of data that a genAI model could be trained upon?
I'll try to deconstruct your question and give you the most informative answer:
Is there a limitation to the amount of data that a genAI model could be trained upon?
In the way that this question is ...
- 2,620
3
votes
Why do we need Upsampling and Downsampling in Progressive Growing of Gans
Use of Transposed Convolution can lead to checkerboard artifacts. So we prefer to up-sample and then apply convolution. You can check this article for more information https://distill.pub/2016/deconv-...
3
votes
Accepted
What is the purpose of the GAN?
GANs were invented in a bar somewhere in Montreal, Canada. At said bar, the idea was that neural networks could be used for generating new examples from an existing distribution. This was the problem:
...
- 3,847
3
votes
How can we process the data from both the true distribution and the generator?
Let's start at the beginning. GANs are models that can learn to create data that is similar to the data that we give them.
When training a generative model other than a GAN, the easiest loss function ...
- 1,186
3
votes
How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?
Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the ...
- 316
3
votes
Accepted
Is Diffusion model instable during the training?
This is (mostly) because of random weight initialization, each time you instance your model, starting weights are different, and during training the model weights converge to a different local minima.
...
- 1,355
3
votes
Accepted
About cosine noise schedule in Diffusion Model
I will suppose that you already have understood how diffusion models work. Some good resources are this blog and the DDPM paper.
If we look at Figure 3 of the paper, we see that in linear schedule the ...
- 400
3
votes
Accepted
Likelihood function for Gaussian Discriminant Analsis
Your understanding is correct. The indicator function ensures that only the term corresponding to the true class $y_i$ contributes, and all other terms become $1$, effectively ignoring them. This is ...
- 3,212
2
votes
How can we process the data from both the true distribution and the generator?
You can treat a combination of z input and x input as a single sample, and you evaluate how well the discriminator performed the ...
- 458
2
votes
Neural network to get input attributes using only the output value
Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output?
It depends:
If you ...
- 32.9k
2
votes
Accepted
What are the possible social consequences of training neural networks with artificially generated data?
We can already observe information bubbles on social media, where the circle is that the ML algorithms learn what content people like and give more similar content based on clicks and so on. From a ...
- 927
2
votes
Accepted
Why do we use $D(x \mid y)$ and not $D(x,y)$ in conditional generative adversarial networks?
It looks like you're asking about the difference between using conditional and joint probabilities.
The joint probability $$D(x,y)$$ is the probability of x and y both happening together.
The ...
- 9,342
Only top scored, non community-wiki answers of a minimum length are eligible