# How do we know if GPT-2 is a better language model?

You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole model fearing dangerous implications. Along the paper, they also published a manifesto to justify their choice: "Better Language Models and Their Implications". And soon a lot of media were publishing articles discussing the choice and its effectiveness to actually prevent bad implications. I am not here to discuss the ethical components of this choice but the actual performance of the model.

The model got my attention too and I downloaded the small model to play with. To be honest I am far from impressed by the results. Some times the first paragraph of the produced text appears to make sense, but nine times out of ten it is giberish by the first or the second sentence. Exemples given in the paper seems to be "Lucky" outputs, cherry picked by human hands. Overall, the paper may suffer from a very strong publication bias.

However most article we can read on the internet seems to take its powerfullness for granted. The MIT technology review wrote : "The language model can write like a human [...]", The Guardian wrote "When used to simply generate new text, GPT2 is capable of writing plausible passages that match what it is given in both style and subject. It rarely shows any of the quirks that mark out previous AI systems, such as forgetting what it is writing about midway through a paragraph, or mangling the syntax of long sentences.". The model appears generally qualified as a "breaktrough". These writings do not match my personnal experimentation as produced texts are rarely consistent / syntaxically correct.

My question is : whitout the release of the whole model for ethical reasons, how do we know if the model is really that powerfull ?

The best way in evaluating the output of the GPT-2 software is to interpret the sentences as brilliant. That means, the role play is about an Artificial Intelligence which is producing first class stories, and the aim of the human writer is to read and criticize these stories. But not in a destructive way but with the aim to help the AI to become a better writer. Example:

GPT-2: Hello World. The grass is green.
human critique: Great, the system can speak.
GPT-2: And the sky is blue.
human critique: Amazing, the next sentence makes also sense. Gimme more.
GPT-2: The tree are blowing at the wind.
human critique: There is a small typo error, but it's no problem.
The GPT-2 will get the next nobel price in literature.


The human critique is not 100% honest. That means, in reality the GPT-2 ability to write text is below a beginner, but nobody cares. It's part of the human-robot-interaction, that the machine is always right. The human critique has to utilize his knowledge about literature, journalism and English language as a helping hand to make the AI feel comfortable. It's not the task of the human to determine the limits of a robot.

• I get there is a huge publication bias... "incoherent langage model has no practical uses" would not attract many clicks. What I don't get is (1) the point for OPENAI of publishing the worse version and (2) the point for journalist to calling it a "breaktrough". I mean I would actively seek (and pay) a media calling people on their bullshit. – lcrmorin Feb 25 at 14:06

You have experimented with a small model (117M parameters). OpenAI has now released the medium sized model (345M parameters). Note that the full model has 1.5B parameters. Also note that GPT-2 has been trained on a tiny fraction of all available text. It's almost guaranteed that a larger model trained on more text will generate better text.

I have experimented with the 345M model, and I agree with The Guardian assessment. See for yourself:

Prompt:

I want to generate beautiful music using machine learning. Do you think a computer will ever be able to compose something as novel as creations of Mozart, Bach, or Chopin?

Results (first four samples, not cherry picked):

Yes, computers will probably be able to do it by 2020. I think the key for us will be to make music that is interesting and beautiful, so we will never really know. My hope would be that computers will become far more efficient over time – something that is already happening in a lot of ways like in big data – while still being able to do more things that they are trained on, like find patterns. I think that's where I'd want to use neural networks in the future.

We don't know, but it will probably happen. That's an interesting question to ask. Of course we have AI, which can do interesting things without human input. It can recognize the world, find objects and images, it understands language, we can make games for it. But if you imagine an AI being able to write music, will it even be good enough? How much better than human players? How much better can someone else do? AI will never really be at its best because it will always have to come up with creative ways to use and play it. To ask a question like it is good enough is to ignore that it is human. You are only allowed to ask good questions

The software will help, as it already can do. But it will also allow more human creators, which is what I think it should do more of to bring music to any genre. In terms of composing, it's a matter of creativity in humans in general. But how does it achieve that? Let's just say that there are fewer humans composing today than ever before. Do you remember the day when, by chance, a computer could write lyrics to Mozart's Sonata No. 4, or the day when a human wrote lyrics to Mozart's La Résistance du Ciel No. 2? That was back in 1960, and it's just not true anymore. As we move forward, we are not going to create music that human musicians cannot compose. What is your goal as a computer music creator?

At the core of this idea is that music is a collection of sounds – it was the very first art form, it was the first music that humans evolved, a collection of sounds. Music is not an abstract thing of sound, but a collection of sounds. You can't write music the way we write and compose books and films, because we write and compose with sounds in mind. This is where machine learning comes in. You can read some books about how to get better music writers or how to improve the quality of their music, but you cannot make a difference with this kind of information at all. What does music teach you? To write and perform beautifully, to convey emotion accurately, and to create lasting and rich relationships with

• 4 random exemples doesn't constitute a representative metric. – lcrmorin Jun 19 at 17:31
• We don't have any good metrics to evaluate the quality of generated text. If you read papers on the topic you will see that human evaluation remains the best indicator of quality. – MichaelSB Jun 19 at 18:06