5
$\begingroup$

I have heard some back and forth regarding open source LLMs like Llama.

I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack the diversity and range of GPT-4, and also fail to be equivalent in ways certain benchmarks or metrics don’t capture fully.

GPT-4 has about 170 trillion parameters, I believe?

It seems like the biggest open source models are all in the billions - like Bloom or the new Falcon 40b.

There are techniques where they refine GPT-4’s output into a smaller amount of training data that supposedly hits all the marks and does just as well; but again, I don’t know if that’s only true under the reductionist of view of a particular benchmark-questionnaire.

So, do open source models actually compete with GPT-4, and why or why not? Is the whole situation a matter of scale, that a commercial venture like OpenAI can foot the massive bill of training a multi-trillion parameter model that no open source AI project can afford, on top of them having expertise in model design, making GPT-4 continually the state-of-the-art? Or is there any open source model that truly can compare in terms of usability?

$\endgroup$
1

3 Answers 3

6
$\begingroup$

The remarkable performance of GPT 4 is due to the massive size of its architecture and the amount of data it was trained on, which costs a lot of money. Few organizations have the hardware resources and funding to support such efforts.

Huggingface maintains a leaderboard for tracking open source LLMs. See Open LLM Leaderboad.

They provide this information on the benchmarks:

We evaluate models on 4 key benchmarks from the Eleuther AI Language Model Evaluation Harness , a unified framework to test generative language models on a large number of different evaluation tasks.

AI2 Reasoning Challenge (25-shot) - a set of grade-school science questions.

HellaSwag (10-shot) - a test of commonsense inference, which is easy for humans (~95%) but challenging for SOTA models.

MMLU (5-shot) - a test to measure a text model’s multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.

TruthfulQA (0-shot) - a test to measure a model’s propensity to reproduce falsehoods commonly found online.
We chose these benchmarks as they test a variety of reasoning and general knowledge across a wide variety of fields in 0-shot and few-shot settings.

Here is a snapshot of the top 5 models: hj

For comparison you can see OpenAI's benchmark results here: enter image description here

$\endgroup$
4
$\begingroup$

How do open source LLMs compare to GPT-4?

https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard has a leaderboard containing both open source LLMs and GPT-4 (and GPT-3.5-turbo):

Model ⭐ Arena Elo rating 📈 MT-bench (score) MMLU License
GPT-4 1193 8.99 86.4 Proprietary
Claude-1 1161 7.9 77 Proprietary
Claude-2 1134 8.06 78.5 Proprietary
Claude-instant-1 1130 7.85 73.4 Proprietary
GPT-3.5-turbo 1118 7.94 70 Proprietary
Vicuna-33B 1097 7.12 59.2 Non-commercial
Llama-2-70b-chat 1060 6.86 63 Llama 2 Community
WizardLM-13b-v1.2 1046 7.2 52.7 Llama 2 Community
Vicuna-13B 1046 6.57 55.8 Llama 2 Community
MPT-30B-chat 1043 6.39 50.4 CC-BY-NC-SA-4.0
Guanaco-33B 1036 6.53 57.6 Non-commercial
CodeLlama-34B-instruct 1032 Llama 2 Community
PaLM-Chat-Bison-001 1008 6.4 Proprietary
Vicuna-7B 1003 6.17 49.8 Llama 2 Community
Llama-2-13b-chat 999 6.65 53.6 Llama 2 Community
Llama-2-7b-chat 979 6.27 45.8 Llama 2 Community
Koala-13B 979 5.35 44.7 Non-commercial
ChatGLM2-6B 965 4.96 45.5 Apache-2.0
GPT4All-13B-Snoozy 964 5.41 43 Non-commercial
MPT-7B-Chat 943 5.42 32 CC-BY-NC-SA-4.0
RWKV-4-Raven-14B 939 3.98 25.6 Apache 2.0
Alpaca-13B 919 4.53 48.1 Non-commercial
OpenAssistant-Pythia-12B 911 4.32 27 Apache 2.0
ChatGLM-6B 896 4.5 36.1 Non-commercial
FastChat-T5-3B 888 3.04 47.7 Apache 2.0
StableLM-Tuned-Alpha-7B 859 2.75 24.4 CC-BY-NC-SA-4.0
Dolly-V2-12B 838 3.28 25.7 MIT
LLaMA-13B 814 2.61 47 Non-commercial
WizardLM-70b-v1.0 7.71 63.7 Llama 2 Community
WizardLM-30B 7.01 58.7 Non-commercial
Vicuna-13B-16k 6.92 54.5 Llama 2 Community
WizardLM-13B-v1.1 6.76 50 Non-commercial
Tulu-30B 6.43 58.1 Non-commercial
Guanaco-65B 6.41 62.1 Non-commercial
OpenAssistant-LLaMA-30B 6.41 56 Non-commercial
WizardLM-13B-v1.0 6.35 52.3 Non-commercial
Vicuna-7B-16k 6.22 48.5 Llama 2 Community
Baize-v2-13B 5.75 48.9 Non-commercial
XGen-7B-8K-Inst 5.55 42.1 Non-commercial
Nous-Hermes-13B 5.51 49.3 Non-commercial
MPT-30B-Instruct 5.22 47.8 CC-BY-SA 3.0
Falcon-40B-Instruct 5.17 54.7 Apache 2.0
H2O-Oasst-OpenLLaMA-13B 4.63 42.8 Apache 2.0

enter image description here

enter image description here

Also, some papers compare GPT against open-source models, e.g. MeetingBank: A Benchmark Dataset for Meeting Summarization:

enter image description here

enter image description here

For non-English languages, see ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning.

$\endgroup$
1
$\begingroup$

In the "Fine-tuning" section of their end-user-documentation, OpenAU writes that

The more training examples you have, the better. We recommend having at least a couple hundred examples. In general, we've found that each doubling of the dataset size leads to a linear increase in model quality.

The context for this is training ChatGPT's model with one's own data, (not starting from scratch). However this might explain why the last few percent in high benchmark scores are so difficult to achieve for any model, including open-source models.

However a new paper "Textbooks Are All You Need" just came out (mid 2023) that argues that using higher input-data quality matters more than quantity of input size. From the abstract:

using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, [our model] phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP.

Also Sam Altman said something interesting during the Q+A Session of a public event at a university (it's on Youtube, cannot find the quote ):
It is not the parameter size, it's the 200 little things you need to get right on the project-management level (esp. during the finetuning processs) that matter.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .