10
$\begingroup$

Mathematically, I wouldn't expect LLMs to be able to reproduce source texts exactly unless the source text was the probable outcome given some prompt. However, I have now tested HuggingFaceH4/zephyr-7b-beta, TheBloke/Llama-2-7B-Chat-GGUF, and ChatGPT and they are all able to accurately reproduce Moby Dick.

HuggingFaceH4/zephyr-7b-beta

Prompt:

Print the first page of Moby Dick.

Response:

I'm unable to print documents directly, but I can provide you with the text of the first page of moby dick by herman melville:

call me ishmael. some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation.  
...SNIP...

TheBloke/Llama-2-7B-Chat-GGUF

print the first page of moby dick
Certainly! Here is the first page of Moby Dick by Herman Melville:
Call me Ishmael. Some years ago—never mind how long precisely—having little or no money in my purse, and nothing particular to interest me on shore, I thought I would sail about a little and see the watery part of the world. It is a way I have of driving off the spleen, and regulating the circulation. Whenever I find myself growing grim about the mouth; whenever it is a damp, drizzly November in my soul; whenever I find myself involuntarily pausing before coffin-makers' shops, and 
...SNIP...

I would have expected them to generate a probabilistic answer but this appears to be exact. Is that simply because it was the likely output? Is there some alternate facility that is producing these results? I used h2ogpt for testing.

Update

It seems like it's probably just that it's mathematically likely because if you ask it other question it does just generate nonsense. Ex: What are the chapters in Moby Dick? I haven't read Moby Dick in English, but I don't recall a chapter 90, "The Whale's Eye Cups"

$\endgroup$
3
  • $\begingroup$ How do you know it's reproducing and not just regurgitating? Bard would say "I'm also able to access and process information from the real world through google search and keep my response consistent with search results." - all it's doing is googling it. Get it to reproduce something that's not part of Project Gutenberg for free and then you might have something. $\endgroup$
    – Mazura
    Jan 7 at 18:07
  • $\begingroup$ @Mazura That's not how h2ogpt works. h2ogpt is a 100% on prem model. Being on-prem and not talking to the Internet is one of the core purposes of the project. $\endgroup$ Jan 8 at 14:33
  • $\begingroup$ So sort of like Watson with the entirety of the internet downloaded; that doesn't really count. It was a part of the "pre-generated databases" you fed it? github.com/h2oai/h2ogpt/blob/main/docs/README_LangChain.md - "A large Wikipedia database is also available." (that is like a 1.5 megs DL; I don't get it. That should be over 20GB worth of files...). Kinda like asking how it knows everything on wiki; because it has wiki downloaded? But if it's like a 2MB exe program with no databases thrown at it and it can do that w/o the internet.... $\endgroup$
    – Mazura
    Jan 12 at 19:42

3 Answers 3

15
$\begingroup$

Google "call me ishmael. some years ago—never mind how long precisely—having" and you'll see a fair number of results. LLM training sets are likely to have several copies of it as well, hence your observation. Note that some LLM training datasets do try to deduplicate data, e.g. https://huggingface.co/datasets/uonlp/CulturaX

$\endgroup$
13
$\begingroup$

LLMs are information-theoretically just very lossy compression of their entire corpora, and are large enough for the "decompression" of parts to be recognizable and reasonably faithful. I don't think there's a really good understanding of which parts are recoverable to this extent. Naively one might think it's only very common material that appears repeatedly in the corpus, but they've also been found to reproduce exact or near-exact text for single-instance personal information. I'm not familiar with it but I suspect this is an exciting research area.

"The probable outcome given some prompt" is one way of looking at what is computationally being done with the model, but it's important to realize that this assumes a probability model, which was derived from the training corpus. It might help as an exercise to imagine doing the model with a naive Markov chain instead, where it's mentally tractable to see how this kind of text can be encoded reproducibly in the probability model.

$\endgroup$
8
  • 7
    $\begingroup$ If the description of GPT-4 in the New York Times lawsuit is correct, it's not even always lossy compression. GPT-4 apparently has enough parameters to losslessly encode a complete copy of its training data at a 10:1 compression ratio, while still having several hundred billion parameters left over. $\endgroup$
    – Mark
    Jan 5 at 23:23
  • 3
    $\begingroup$ Those remaining parameters could allow for alternate paths in the stored text chain to provide variety and context matching. I got an amazing response once: "translation provided by [real company name here]". It looked like a direct leak of training data. It came as a response of a failed input, some breath in speech input to ChatGPT that was transcribed as empty text. $\endgroup$ Jan 6 at 8:20
  • 1
    $\begingroup$ Years ago, a technique for text to speech was "unit selection". A raw speech database was labeled and optimum path chosen according to phonetic transcription and prosody. A similar approach could be thought of for text generation. $\endgroup$ Jan 6 at 10:09
  • $\begingroup$ @JaumeOliverLafont Note: such responses are quite common from non-Chat GPTs, since they are only playing a game of "predict the next word". I don't know - and perhaps nobody outside of OpenAI knows - exactly what special sauce they wrapped around a predict-the-next-word game to turn it into a good question and answer machine, but it's not crazy that sometimes you see some of its inner nature peek through. $\endgroup$
    – user253751
    Jan 6 at 19:09
  • 1
    $\begingroup$ @R..GitHubSTOPHELPINGICE That was probably true of the first few versions but now it seems to be much more sophisticated. There's also this RLHF thing I don't really understand. $\endgroup$
    – user253751
    Jan 6 at 20:00
-1
$\begingroup$

When we create and train neural networks, the goal is to get them to model a general representation of the input data, so that they will produce the desired output for inputs that are generally similar to inputs that were in the training data.

But if your network has too many parameters, it doesn't need to generalize during the training stage. It might instead use those parameters to instead store representations of the input data and generate the desired output precisely - basically storing input/output pairs for everything input/output pair that's present in the training data. That won't necessarily happen, but it can happen. If you give it enough parameters, and train it long enough, it probably will happen.

And then you feed it something that wasn't in the training set, and it produces garbage because it didn't generalize at all, it just encoded the inputs and outputs that it was trained with.

This is one of the reasons that it's a good idea to train with one set of input/output pairs, and test with another set. If the network is generalizing, it will do a pretty good job with the test set even though those input/output pairs aren't part of the training set. But if the network is memorizing, it will do a pretty terrible job with the test set, because those input/output pairs were not memorized.

We generally* want the network to have the smallest number of parameters that will give acceptably good results. If you constrain the network to have a small number of parameters, it can't just encode the inputs and outputs verbatim - to perform well on the test set, it pretty much needs to encode a generalization of the training data, rather than the specifics of the training data.

In the context of the "predict the next word" approach used with LLMs, memorizing input/output pairs basically means that when prompted with a fragment of training data, it will regurgitate the remainder of that training data.

So, if an LLM reproduces lots of training data verbatim, that's probably a pretty good sign that the model has far too many parameters. A model with fewer parameters can probably perform just as well (probably better) by generalizing more and memorizing less.

* All generalizations are false, but some generalizations are useful.

$\endgroup$
2
  • 1
    $\begingroup$ This response isn't inherently bad, but it is misleading if you don't already know what you're looking at. Suggest rewriting it. Eliminate irrelevant language about irritating practitioners, reword so that it's clear that having too parameters doesn't definitely lead to overfitting (also as I mentioned in my update ChatGPT can't reproduce the entire book so suggest clarifying your point here), smaller networks do generally avoid overfitting but this is all highly dependent on use case, avoid saying generally and just give the specific scenarios where you think that's true $\endgroup$ Jan 9 at 14:59
  • $\begingroup$ I moved the disclaimer into a footnote and clarified that memorization is not inevitable. The question "Why are LLMs able to reproduce bodies of known text exactly?" is asking a general question about LLMs, so I don't think it would be useful to remove the generalizations. $\endgroup$
    – NSFW
    Jan 10 at 20:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .