15 votes

Why are LLMs able to reproduce bodies of known text exactly?

Google "call me ishmael. some years ago—never mind how long precisely—having" and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, ...
Franck Dernoncourt's user avatar
13 votes

Why are LLMs able to reproduce bodies of known text exactly?

LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the "decompression" of parts to be recognizable and reasonably faithful. I ...
R.. GitHub STOP HELPING ICE's user avatar
11 votes
Accepted

Is plain autoencoder a generative model?

An autoencoder is not considered a generative model, because it only reconstructs the given input. You could use the decoder like a generative model by putting in different vectors. However, the ...
Chillston's user avatar
  • 1,674
9 votes
Accepted

What is the meaning of $V(D,G)$ in the GAN objective function?

To understand this equation first you need to understand the context in which it is first introduced. We have two neural networks (i.e. $D$ and $G$) that are playing a minimax game. This means that ...
Djib2011's user avatar
  • 3,183
7 votes
Accepted

How can we process the data from both the true distribution and the generator?

The Focus of This Question "How can ... we process the data from the true distribution and the data from the generative model in the same iteration? Analyzing the Foundational Publication In the ...
Douglas Daseeco's user avatar
5 votes

Are deep learning models suitable for training with sparse data?

The problem isn't the GAN but the implementation of its discriminator which is typically a convolutional neural network (CNN). CNNs have trouble with sparse data. They require dense data to learn ...
Brian O'Donnell's user avatar
5 votes

Why diffusion model always use U-Net?

I don't have a definitive answer but I'd state my intuitions anyways: Diffusion models are highly related to the idea of stacked denoising autoencoders [Kumar et al. (2014)]. Additionally, U-Net-like ...
Chillston's user avatar
  • 1,674
4 votes

Why is the variational auto-encoder's output blurred, while GANs output is crisp and has sharp edges?

The key is: VAE usually use a small latent dimension, the information of input is so hard to pass through this bottleneck, meanwhile it tries to minimize the loss with the batch of input data, you ...
seasonyc's user avatar
4 votes
Accepted

Other deep learning image generation techniques besides GANs?

There are several generative models that have been proposed before or roughly at the same time of the GAN (2014). For example, the deep Boltzman machine (2009), deep generative stochastic network (...
nbro's user avatar
  • 40.2k
4 votes
Accepted

How can we find find the input image which maximizes the class-probability for an ANN?

In deep networks there is actually a wide variety of solutions to the problem, but if you need to find one, any easy way to do this is just through normal optimization schemes $$\hat x = argmin_x \ L(...
mshlis's user avatar
  • 2,359
4 votes

How can we find find the input image which maximizes the class-probability for an ANN?

Probably the simplest way to search for an image with the highest probability of being a cat is to use a technique similar to Deep Dream: Load the network for training, but freeze all the network ...
Neil Slater's user avatar
  • 31.5k
4 votes
Accepted

Where is the mistake in my derivation of the GAN loss function?

I guess the issue is you lost track of where the samples came from and since you requested a math explanation I'll try to go step by step using my notation and without checking other material to avoid ...
Nicola Bernini's user avatar
4 votes
Accepted

Would it be possible to determine the dataset a neural network was trained on?

You can already do this with some neural networks, such as GANs and VAEs, which are generative models that learn a probability distribution over the inputs, so they learn how to produce e.g. images ...
nbro's user avatar
  • 40.2k
4 votes

What makes ChatGPT a generative model?

What are generative (and discriminative) models? If the model learns a distribution of the form $p(x)$ or $p(x, y)$, where $x$ are the inputs and $y$ the outputs/labels, from which you can sample data,...
nbro's user avatar
  • 40.2k
4 votes
Accepted

What's the architecture that allows the generation of new images based on input image in tools like Midjourney?

Midjourney (and DALL-E 2 I think) uses a concept vector (or "embedding") to condition its image outputs, which can be produced in at least two ways: By summarising text input By converting ...
Neil Slater's user avatar
  • 31.5k
4 votes

Is there a limitation to the amount of data that a genAI model could be trained upon?

I'll try to deconstruct your question and give you the most informative answer: Is there a limitation to the amount of data that a genAI model could be trained upon? In the way that this question is ...
Robin van Hoorn's user avatar
3 votes

Why do we need Upsampling and Downsampling in Progressive Growing of Gans

Use of Transposed Convolution can lead to checkerboard artifacts. So we prefer to up-sample and then apply convolution. You can check this article for more information https://distill.pub/2016/deconv-...
Giannis Kousis's user avatar
3 votes
Accepted

What is the purpose of the GAN?

GANs were invented in a bar somewhere in Montreal, Canada. At said bar, the idea was that neural networks could be used for generating new examples from an existing distribution. This was the problem: ...
Jaden Travnik's user avatar
3 votes

How can we process the data from both the true distribution and the generator?

Let's start at the beginning. GANs are models that can learn to create data that is similar to the data that we give them. When training a generative model other than a GAN, the easiest loss function ...
Seth Simba's user avatar
  • 1,186
3 votes

Book(s) on generative models

From the theoretical foundations one can look into the Chapter 20: Deep Generative Models of the classic DL book by Goodfellow, Bengio https://amzn.to/2MmZNbH. Not ...
spiridon_the_sun_rotator's user avatar
3 votes
Accepted

How to determine the quality of synthetic data?

Due to subjective nature, quantitative evaluation of synthetic images is difficult in general. However, there are metrics like Inception Score or FID score that are used for evaluation of generative ...
ayandas's user avatar
  • 238
3 votes
Accepted

Why don't we also need to approximate $p(x \mid z)$ in the VAE?

What I can guess here is that, in VAEs, we assume $p(z)$ (prior), so we are able to calculate $p(x \mid z)$, but for $p(x)$ we can't assume its distribution? Is it right? You could assume $p(x)$ is ...
nbro's user avatar
  • 40.2k
3 votes

How to generate new data given a trained VAE - sample from the learned latent space or from multivariate Gaussian?

Few more clarifications. While the correct thing to do is draw from the prior, we have no guarantees that the aggregated posterior will cover the prior. Think of the aggregated posterior as the ...
sfotiadis's user avatar
  • 181
3 votes
Accepted

Is Diffusion model instable during the training?

This is (mostly) because of random weight initialization, each time you instance your model, starting weights are different, and during training the model weights converge to a different local minima. ...
Dr. Snoopy's user avatar
  • 1,320
2 votes

How can we process the data from both the true distribution and the generator?

You can treat a combination of z input and x input as a single sample, and you evaluate how well the discriminator performed the ...
Alpha's user avatar
  • 458
2 votes

Neural network to get input attributes using only the output value

Now I have a network trained to get an output value from an random set of attributes but, can I use this trained network to get the input attributes using only the desired output? It depends: If you ...
Neil Slater's user avatar
  • 31.5k
2 votes
Accepted

What are the possible social consequences of training neural networks with artificially generated data?

We can already observe information bubbles on social media, where the circle is that the ML algorithms learn what content people like and give more similar content based on clicks and so on. From a ...
mico's user avatar
  • 937
2 votes
Accepted

Why do we use $D(x \mid y)$ and not $D(x,y)$ in conditional generative adversarial networks?

It looks like you're asking about the difference between using conditional and joint probabilities. The joint probability $$D(x,y)$$ is the probability of x and y both happening together. The ...
John Doucette's user avatar
2 votes
Accepted

How can AI be used to more reliably analyze and plan around the tie between climate and emissions?

Can AI provide a more reliable analysis of the gross effects of carbon emissions on extinctions of species ice-cap melting, and other effects? Yes. The work of Judea Pearl and others over the last 20 ...
John Doucette's user avatar
2 votes

Which approach can I use to generate text based on multiple inputs?

Generally, text generators work by modeling the joint distribution of the text by its Bayesian forward decomposition $ \begin{align*} p(w_1, w_2, ..., w_n) &= p(w_1) * p(w_2|w_1) * p(w_3|w_2, ...
mshlis's user avatar
  • 2,359

Only top scored, non community-wiki answers of a minimum length are eligible