How do we know if GPT-2 is a better language model?

Question

You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole model fearing dangerous implications. Along the paper, they also published a manifesto to justify their choice: "Better Language Models and Their Implications". And soon a lot of media were publishing articles discussing the choice and its effectiveness to actually prevent bad implications. I am not here to discuss the ethical components of this choice but the actual performance of the model.

The model got my attention too and I downloaded the small model to play with. To be honest I am far from impressed by the results. Some times the first paragraph of the produced text appears to make sense, but nine times out of ten it is giberish by the first or the second sentence. Exemples given in the paper seems to be "Lucky" outputs, cherry picked by human hands. Overall, the paper may suffer from a very strong publication bias.

However, most article we can read on the internet seems to take its powerfulness for granted. The MIT technology review wrote:

The language model can write like a human

The Guardian wrote

When used to simply generate new text, GPT2 is capable of writing plausible passages that match what it is given in both style and subject. It rarely shows any of the quirks that mark out previous AI systems, such as forgetting what it is writing about midway through a paragraph, or mangling the syntax of long sentences.

The model appears generally qualified as a "breakthrough". These writings do not match my personal experimentation as produced texts are rarely consistent / syntactically correct.

My question is: without the release of the whole model for ethical reasons, how do we know if the model is really that powerful?

Answer 1

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.

I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:

Prompt:

I want to generate beautiful music using machine learning. Do you think a computer will ever be able to compose something as novel as creations of Mozart, Bach, or Chopin?

Results (first four samples, not cherry picked):

Yes, computers will probably be able to do it by 2020. I think the key for us will be to make music that is interesting and beautiful, so we will never really know. My hope would be that computers will become far more efficient over time – something that is already happening in a lot of ways like in big data – while still being able to do more things that they are trained on, like find patterns. I think that's where I'd want to use neural networks in the future.

---

We don't know, but it will probably happen. That's an interesting question to ask. Of course we have AI, which can do interesting things without human input. It can recognize the world, find objects and images, it understands language, we can make games for it. But if you imagine an AI being able to write music, will it even be good enough? How much better than human players? How much better can someone else do? AI will never really be at its best because it will always have to come up with creative ways to use and play it. To ask a question like it is good enough is to ignore that it is human. You are only allowed to ask good questions

---

The software will help, as it already can do. But it will also allow more human creators, which is what I think it should do more of to bring music to any genre. In terms of composing, it's a matter of creativity in humans in general. But how does it achieve that? Let's just say that there are fewer humans composing today than ever before. Do you remember the day when, by chance, a computer could write lyrics to Mozart's Sonata No. 4, or the day when a human wrote lyrics to Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's just not true anymore. As we move forward, we are not going to create music that human musicians cannot compose. What is your goal as a computer music creator?

---

At the core of this idea is that music is a collection of sounds – it was the very first art form, it was the first music that humans evolved, a collection of sounds. Music is not an abstract thing of sound, but a collection of sounds. You can't write music the way we write and compose books and films, because we write and compose with sounds in mind. This is where machine learning comes in. You can read some books about how to get better music writers or how to improve the quality of their music, but you cannot make a difference with this kind of information at all. What does music teach you? To write and perform beautifully, to convey emotion accurately, and to create lasting and rich relationships with