I have heard some back and forth regarding open source LLMs like Llama.
I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack the diversity and range of GPT-4, and also fail to be equivalent in ways certain benchmarks or metrics don’t capture fully.
GPT-4 has about 170 trillion parameters, I believe?
It seems like the biggest open source models are all in the billions - like Bloom or the new Falcon 40b.
There are techniques where they refine GPT-4’s output into a smaller amount of training data that supposedly hits all the marks and does just as well; but again, I don’t know if that’s only true under the reductionist of view of a particular benchmark-questionnaire.
So, do open source models actually compete with GPT-4, and why or why not? Is the whole situation a matter of scale, that a commercial venture like OpenAI can foot the massive bill of training a multi-trillion parameter model that no open source AI project can afford, on top of them having expertise in model design, making GPT-4 continually the state-of-the-art? Or is there any open source model that truly can compare in terms of usability?