# What exactly are the “parameters” in GPT-3's 175 billion parameters and how are they chosen/generated?

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.

Even tutorial sites like this one start talking about the usual parameters, but also say "model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights". So are the 175 billion "parameters" just neural weights? Why then are they called parameters? GPT3's paper shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each "parameter" is just a representation of the encoders or decoders?

An excerpt from this website shows tokens:

In this case, there are two additional parameters that can be passed to gpt2.generate(): truncate and include_prefix. For example, if each short text begins with a <|startoftext|> token and ends with a <|endoftext|>, then setting prefix='<|startoftext|>', truncate=<|endoftext|>', and include_prefix=False, and length is sufficient, then gpt-2-simple will automatically extract the shortform texts, even when generating in batches.

So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the "parameters" are auto-generated somehow.

The attention-based paper mentions the query-key-value weight matrices as the "parameters". Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?

• Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters – David Ireland Jul 26 '20 at 19:39
• Thank you David. So now my understanding is that GPT3 has 96 layers and 175 billion nodes (weights or parameters) arranged in various ways as part of the transformer model. – Nav Jul 27 '20 at 2:35
• It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean). – David Ireland Jul 27 '20 at 8:48
• Understood. The weights are vectors which are called the "learned parameters". Thank you very much, David. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer. – Nav Jul 28 '20 at 5:40