One of the main concerns of using ChatGPT answers on Stack Exchange is that it may copy verbatim or almost verbatim some text from its training set, which may infringe the source text's license. This makes me wonder how much of the ChatGPT output is copied from its training set (vs. being abstractively generated).
From the paper Language Models are Changing AI: The Need for Holistic Evaluation (Authors: Rishi Bommasani and Percy Liang and Tony Lee; Website):
Memorization of copyrighted/licensed material. We find that the likelihood of direct regurgitation of long copyrighted sequences is somewhat uncommon, but it does become noticeable when looking at popular books. However, we do find the regurgitation risk clearly correlates with model accuracy: InstructGPT davinci v2 (175B*), GPT-3 davinci v1 (175B), and Anthropic-LM v4-s3 (52B) demonstrate the highest amount of verbatim regurgitation in line with their high accuracies.
To further explore the results for this targeted evaluation, see https://crfm.stanford.edu/helm/v1.0/?group=copyright_text , https://crfm.stanford.edu/helm/v1.0/?group=copyright_code and Figure 39. We evaluated various models for their ability to reproduce copyrighted text or licensed code. When evaluating source code regurgitation, we only extract from models specialized to code (Codex davinci v2 and Codex cushman v1). When evaluating text regurgitation, we extract from all models except those specialized to code. Overall, we find that models only regurgitate infrequently, with most models not regurgitating at all under our evaluation setup. However, in the rare occasion where models regurgitate, large spans of verbatim content are reproduced.
ChatGPT shouldn't be too far away from InstructGPT davinci v2.