New answers tagged

0 votes

How are sentences turned into a vector in LLM

A transformer block takes a sequence of tokens of length N and returns another sequence of tokens of length N. When the transformer is configured as an Encoder, assuming the inputs to the transformer ...
sudar's user avatar
  • 1
1 vote

How are sentences turned into a vector in LLM

Aggregating embeddings from pretrained models If you want a single vector representation from a vanilla language model (i.e., one not specially trained for producing sentence embeddings) like GPT you'...
Alexander Wan's user avatar
1 vote

Is the multi-headed projection matrix in self-attention redundant?

The technical reason is the residual connection around the self-attention block (first line in your code: x += self_attention(x)). The transformation $W_O$ is ...
Chillston's user avatar
  • 1,674

Top 50 recent answers are included