2
$\begingroup$

What should we keep hidden dimension/embedding dimension (d_model as per attention is all you need paper), greater, equal, or smaller to the context length (n)?

Is there any such relationship between the embedding dimension and context length?

How will it affect the LLM?

$\endgroup$

1 Answer 1

0
$\begingroup$

Embedding Dimension (d_model):

  • One word/token will be represented as "d_model" different numebrs. Larger the d_model,higher thwe capacityof storing the true meaning of the word
  • It also affects the computational complexity of the model. A larger d_model means the model can potentially learn more complex representations but at the cost of increased computation.

Context Length(n):

  • maximum number of previous words/tokens that the model considers when predicting the next word/token.
  • It determines how much “memory” the model has. A larger context length allows the model to take into account more of the previous text, which can be beneficial for tasks that require understanding of longer passages.

There isn’t a direct relationship where changing one would necessitate a change in the other. However, both parameters would impact the overall computational requirements and performance of the model.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .