One of the main concerns of using ChatGPT answers on Stack Exchange is that it may copy verbatim or almost verbatim some text from its training set, which may infringe the source text's license. This makes me wonder how much of the ChatGPT output is copied from its training set (vs. being abstractively generated).

  • 1
    $\begingroup$ Is there any evidence for this "it may copy verbatim or almost verbatim some text from its training set"? This may be true, but I am wondering if there is any evidence. I never tried it, so I don't know. $\endgroup$
    – nbro
    Dec 21, 2022 at 12:55
  • 1
    $\begingroup$ I think this is incorrect. The real problem is that it's generating text based on the probability that a certain token appears in a given position given all of the other tokens so far... which means that the result might be interesting, but isn't necessarily anything like accurate. $\endgroup$ Dec 21, 2022 at 13:23
  • $\begingroup$ @nbro I don't know any. I haven't tried much. $\endgroup$ Dec 21, 2022 at 14:04
  • $\begingroup$ @DavidHoelzer that's indeed another concern. meta.stackexchange.com/q/384410/178179 ; meta.stackexchange.com/q/384652/178179 $\endgroup$ Dec 21, 2022 at 14:04
  • $\begingroup$ This is impossible to answer since the training set is not public. $\endgroup$
    – Dr. Snoopy
    Dec 22, 2022 at 10:25

1 Answer 1


From the paper Language Models are Changing AI: The Need for Holistic Evaluation (Authors: Rishi Bommasani and Percy Liang and Tony Lee; Website):

Memorization of copyrighted/licensed material. We find that the likelihood of direct regurgitation of long copyrighted sequences is somewhat uncommon, but it does become noticeable when looking at popular books. However, we do find the regurgitation risk clearly correlates with model accuracy: InstructGPT davinci v2 (175B*), GPT-3 davinci v1 (175B), and Anthropic-LM v4-s3 (52B) demonstrate the highest amount of verbatim regurgitation in line with their high accuracies.


To further explore the results for this targeted evaluation, see https://crfm.stanford.edu/helm/v1.0/?group=copyright_text , https://crfm.stanford.edu/helm/v1.0/?group=copyright_code and Figure 39. We evaluated various models for their ability to reproduce copyrighted text or licensed code. When evaluating source code regurgitation, we only extract from models specialized to code (Codex davinci v2 and Codex cushman v1). When evaluating text regurgitation, we extract from all models except those specialized to code. Overall, we find that models only regurgitate infrequently, with most models not regurgitating at all under our evaluation setup. However, in the rare occasion where models regurgitate, large spans of verbatim content are reproduced.

ChatGPT shouldn't be too far away from InstructGPT davinci v2.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .