26
$\begingroup$

When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that they might just be sentences.

enter image description here

Even tutorial sites like this one start talking about the usual parameters, but also say "model_name: This indicates which model we are using. In our case, we are using the GPT-2 model with 345 million parameters or weights". So are the 175 billion "parameters" just neural weights? Why then are they called parameters? GPT3's paper shows that there are only 96 layers, so I'm assuming it's not a very deep network, but extremely fat. Or does it mean that each "parameter" is just a representation of the encoders or decoders?

enter image description here

An excerpt from this website shows tokens:

In this case, there are two additional parameters that can be passed to gpt2.generate(): truncate and include_prefix. For example, if each short text begins with a <|startoftext|> token and ends with a <|endoftext|>, then setting prefix='<|startoftext|>', truncate=<|endoftext|>', and include_prefix=False, and length is sufficient, then gpt-2-simple will automatically extract the shortform texts, even when generating in batches.

So are the parameters various kinds of tokens that are manually created by humans who try to fine-tune the models? Still, 175 billion such fine-tuning parameters is too high for humans to create, so I assume the "parameters" are auto-generated somehow.

The attention-based paper mentions the query-key-value weight matrices as the "parameters". Even if it is these weights, I'd just like to know what kind of a process generates these parameters, who chooses the parameters and specifies the relevance of words? If it's created automatically, how is it done?

Improve this question
$\endgroup$
6
  • 5
    $\begingroup$ Parameters is a synonym for weights, which are what you learn. Batch size, learning rate etc are typically hyper parameters $\endgroup$
    – David
    Jul 26, 2020 at 19:39
  • 4
    $\begingroup$ It won’t have 175million nodes, if you think of a simpler neural network then the number of parameters is how many connections there are between nodes. If there was a NN with 1 layer of 2 neurons followed by a layer with 3 neurons, then an output layer with 1 neuron there would be 9 parameters (you could draw this out to see what I mean). $\endgroup$
    – David
    Jul 27, 2020 at 8:48
  • 1
    $\begingroup$ Understood. The weights are vectors which are called the "learned parameters". Thank you very much, David. If you post it as an answer, I'll accept it. Hopefully, this would be of help to anyone else searching for this answer. $\endgroup$
    – Nav
    Jul 28, 2020 at 5:40
  • 1
    $\begingroup$ Note that @DavidIreland is assuming a fully connected NN for 9 parameters $\endgroup$
    – Jordan Morris
    Mar 20 at 9:05
  • 1
    $\begingroup$ and no biases, if you’re being pedantic @JordanMorris $\endgroup$
    – David
    Mar 22 at 1:04

1 Answer 1

Reset to default
23
$\begingroup$

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas weights are what the learning algorithm will learn through training.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .