When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.

enter image description here

Even tutorial sites like this one start talking about the usual parameters, but also say "model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights". So are the 175 billion "parameters" just neural weights? Why then are they called parameters? GPT3's paper shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each "parameter" is just a representation of the encoders or decoders?

enter image description here

An excerpt from this website shows tokens:

In this case, there are two additional parameters that can be passed to gpt2.generate(): truncate and include_prefix. For example, if each short text begins with a <|startoftext|> token and ends with a <|endoftext|>, then setting prefix='<|startoftext|>', truncate=<|endoftext|>', and include_prefix=False, and length is sufficient, then gpt-2-simple will automatically extract the shortform texts, even when generating in batches.

So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the "parameters" are auto-generated somehow.

The attention-based paper mentions the query-key-value weight matrices as the "parameters". Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?

  • 2
    $\begingroup$ Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters $\endgroup$ – David Ireland Jul 26 '20 at 19:39
  • $\begingroup$ Thank you David. So now my understanding is that GPT3 has 96 layers and 175 billion nodes (weights or parameters) arranged in various ways as part of the transformer model. $\endgroup$ – Nav Jul 27 '20 at 2:35
  • $\begingroup$ It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean). $\endgroup$ – David Ireland Jul 27 '20 at 8:48
  • $\begingroup$ Understood. The weights are vectors which are called the "learned parameters". Thank you very much, David. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer. $\endgroup$ – Nav Jul 28 '20 at 5:40

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.