# Was ChatGPT trained on Stack Overflow data?

Has ChatGPT used highly rated and upvoted questions/answers from Stack Overflow in its training data?

For me it makes complete sense to take answers that have upwards of 100 upvotes and include them in your training data, but people around me seem to think this hypothesis doesn't make sense. Is there a way to confirm this?

• It's worth noting that even with the option you suggest, the answers produced by ChatGPT wouldn't be more correct. Maybe they would be slightly better written, but this is not the issue. Jan 10 at 16:30
• Think about the attrition this inbreeding will cause to the quality of SO answers.. People ask ChatGPT for solutions which come in part from SO, then they post the answers to SO, ChatGPT learns from these answers but doesn't get them perfect, so the next time someone asks ChatGPT for a similar answer, the result is a little more incorrect, and is then again posted to SO.. Jan 11 at 9:56
• @tplive. That's exactly the reason SO has a feedback system. In my experience, the answers supposedly knowledgeable people post are often no walk in the park. Jan 11 at 18:28
• It certainly didn't learn from stackexchange to apologize for its mistakes. Jan 24 at 9:40
• It would possibly make more sense to use Accepted answers rather than the most upvoted ones. Numbers of upvotes - especially on Stack Overflow (as opposed to other SE sites) - are more reflective of the number of people who wanted to know the answer than the quality of the answer. There are some very good answers with few upvotes simply because it's not a common problem. Mar 21 at 12:22

ChatGPT is in the Large Language Models (LLM) category. The most (in)famous GPT model is probably GPT-3, because since then, researchers realized that LLMs mostly follow a predictable scaling law, thus the more data and the bigger model, the better. It is accurate to say that ChatGPT was trained with Stack Overflow data, but it should be all Stack Overflow instead of just most upvoted answers/comments.

The Wikipedia page of GPT-3 and their paper mentions that GPT-3 was trained on multiple datasets, and one of which is the Common Crawl, which basically crawls everything on the Internet. Some data pre-processing was done before training, but the authors did not mention removing the comments, so we can say that it is all Stack Overflow data.

If we look at the Common Crawl data in Sep 2022, there is indeed the domain com.stackoverflow in their list. Thus, while ChatGPT was trained on Stack Overflow data, it is trained on all Stack Overflow data instead of just most upvoted answers.

However, if you think ChatGPT's code output is of high quality, think again, because Stack Overflow temporarily bans ChatGPT because the average rate of getting correct answers from ChatGPT is too low, the posting of answers created by ChatGPT is substantially harmful to the site and to users who are asking and looking for correct answers (quoted from the link).

Here is the justification of them:

The primary problem is that while the answers which ChatGPT produces have a high rate of being incorrect, they typically look like they might be good and the answers are very easy to produce. There are also many people trying out ChatGPT to create answers, without the expertise or willingness to verify that the answer is correct prior to posting. Because such answers are so easy to produce, a large number of people are posting a lot of answers. The volume of these answers (thousands) and the fact that the answers often require a detailed read by someone with at least some subject matter expertise in order to determine that the answer is actually bad has effectively swamped our volunteer-based quality curation infrastructure.

EDIT: a comment below also confirmed that GPT-2, the predecessor of GPT-3 and ChatGPT, was trained with Stack Overflow data.

• How do you know it was trained on all data? The crawler might have accessed all data. But how do we know this data hasn't been preprocessed? E.g. remove comments etc. Jan 9 at 14:32
• @ThomasWeller See Appendix A of the paper. I don't think they removed comments. Jan 9 at 17:02
• Note that because the average rate of getting correct answers from ChatGPT is too low is not sourced in the link you've provided. Jan 10 at 0:39
• I would argue that (at least in the general population), ChatGPT is now the most famous GPT model, by quite a large margin. Jan 10 at 9:55
• It is also known that GPT2 was trained on Stack Exchange, and Stack Overflow. Jan 11 at 1:39