0
$\begingroup$

For example, would a Large Language Model (LLM) with parameter size 140 Billion have 140 Billion dimensions as defined in deep learning as the number of nodes in the input layer?

Another way to ask this might be: Is 140B parameters the same as saying 140B nodes/dimensions in the encoder of the LLM?

$\endgroup$

1 Answer 1

0
$\begingroup$

Not in terms of the definition in the linked post. That kind of dimensionality would refer to the dimensionality of the input.

As discussed in post, there's also some ambiguity here, as data augmentation can change the dimensionality of the input. You could consider the dimensionality of the text input (although it might be hard to quantify the dimensionality of strings). You could consider the dimensionality of the text embeddings, which would be max_input_length x embedding_dim.

Both of these are not necessarily related to the number of parameters. (E.g., you can have a model with 130b parameters but only takes inputs with 10 tokens or 1000 tokens)

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .