4
$\begingroup$

Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it be possible for a company like Meta or Google to have a model equal to ChatGPT/GPT3.5 in the next month or so? Why or why not?

I understand that a big language model is expensive to train, so I'm expecting only large companies to be able to train such models to a sufficient extent.

Improve this question
$\endgroup$
8
  • $\begingroup$ Can you provide more details about "oh its challenging Google" and about the articles and videos that claim that? It's challenging but most likely not because they don't have the computation power, but for other reasons, which are probably not true. $\endgroup$
    – nbro
    Jan 25 at 9:28
  • $\begingroup$ Im not interested in whether or not the articles or videos are true but only whether it is difficult for others to just replicate their work and what the reasons are for that. I removed the whole 'videos and articles'. $\endgroup$
    – Robin van Hoorn
    Jan 25 at 10:13
  • $\begingroup$ Ok. I think now the question in the body is more objective. I would also change the title then. Another thing is: if you focus on the ChatGPT, then people could simple answer by saying that it's difficult because there's still no research paper available. Maybe you're interested just in ChatGPT or maybe in GPT3. I don't know. Make it clear if you're just interested in Google or other companies and even "normal people" $\endgroup$
    – nbro
    Jan 25 at 10:47
  • $\begingroup$ Generalized it further ;) Thanks for the input $\endgroup$
    – Robin van Hoorn
    Jan 25 at 10:55
  • 3
    $\begingroup$ Are you aware that Google made Lamda? blog.google/technology/ai/lamda $\endgroup$
    – Dr. Snoopy
    Jan 25 at 12:46

2 Answers 2

Reset to default
8
$\begingroup$

Challenges to reproduce ChatGPT:

  • Compute cost
  • Collect training data
  • Find the proper choice for network architecture + RL (OpenAI hasn't published all the details)

Two interesting papers on training cost vs. LLM quality:

For some tasks, "smaller LLMs" can perform well e.g. see Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto. Benchmarking Large Language Models for News Summarization. arXiv:2301.13848.:

We find instruction tuning, and not model size, is the key to the LLM’s zero-shot summarization capability

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Indeed, compute cost is not trivial! Some estimates place the cost of training the model once at several million dollars, let alone the personnel costs, costs for data-gathering, and cost of any kind of hyper-parameter tuning or architecture search. $\endgroup$
    – Sycorax
    Feb 21 at 23:52
  • $\begingroup$ Also, the human cost of having someone manually label the data. $\endgroup$
    – Anshuman Kumar
    Mar 29 at 9:07
  • $\begingroup$ @AnshumanKumar yes I was including it in collect training data. There are some open access data eg C4 + github.com/nomic-ai/gpt4all $\endgroup$
    – Franck Dernoncourt
    Mar 29 at 9:15
2
$\begingroup$

Actually, Google created a bigger model than GPT-3 and models in the GPT-3.5 series, and consequently ChatGPT too (because ChatGPT is based on a GPT-3.5 model) - Switch-C has trillions of parameters, one order of magnitude bigger than the GPT models that I know of, and it was developed before ChatGPT was announced. I don't know how many parameters ChatGPT has exactly, but it shouldn't have more than several billions of parameters.

So, what makes reproducing a model like ChatGPT difficult for companies like Google? Definitely, not the lack of computational resources or money, but the lack of transparency. My impression is that Google also tends to be open-source, as opposed to OpenAI, which wants to make money of everything.

Moreover, I'd like to note that the GPT models have received a lot of hype, but there are other pre-trained models (e.g. Lambda or Switch-C), for example, developed by Google, that maybe should also have our attention. Google simply doesn't need to generate all this hype to get the money, as they still get most of their revenue from ads (the last time I checked)

Improve this answer
$\endgroup$
12
  • $\begingroup$ There is something I'm not sure of and Icant find an answer even in OpenAI's blog. Is GPT-3.5 (series) a complete separate model thas has been trained from scratch or is it a fine tuned version of GPT-3 for code-completion and conversations ? $\endgroup$
    – iMad
    Feb 2 at 13:13
  • 1
    $\begingroup$ @iMad The only reliable info that I found about this topic was here: platform.openai.com/docs/model-index-for-researchers. Maybe it's not accurate to say that OpenAI claims that all models in this series are more capable. I should rewrite that part of my answer $\endgroup$
    – nbro
    Feb 2 at 13:39
  • $\begingroup$ Actually I wasn't referring to your answer specifically, it's just that I can't find any reliable information about how GPT-3.5 has been trained, its differences with GPT-3 etc. In OpenAI's docs, it's mentioned that InstructGPT is somehow a fine-tuned version of GPT-3, and that ChatGPT was trained in the same way as InstructGPT, but no explicit claim about the relationship between the GPT-3 base model and the GPT-3.5 series. $\endgroup$
    – iMad
    Feb 2 at 16:24
  • $\begingroup$ I think you need to back up that middle paragraph. AFAIK, OpenAI publish papers on their models, and explain how ChatGPT was trained here - openai.com/blog/chatgpt (also see links to their publications on Arxiv). Google's LLMs are not open source. So I am not seeing a difference in transparency between the two companies regarding these kind of products, unless you can provide more information $\endgroup$
    – Neil Slater
    Feb 22 at 9:25
  • $\begingroup$ @NeilSlater I'm saying that my impression is that Google tends to be open-source, while OpenAI doesn't. My impression could not be aligned with the reality. I'm not saying that Google is transparent about everything or more transparent than OpenAI about anything. I'm saying that what makes reproducing a model like ChatGPT is the lack of transparency. I'm also not saying that there isn't info about ChatGPT. I'm saying there isn't enough info about everything in order to reproduce it exactly. $\endgroup$
    – nbro
    Feb 22 at 11:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .