New answers tagged

0 votes

How are sentences turned into a vector in LLM

A transformer block takes a sequence of tokens of length N and returns another sequence of tokens of length N. When the transformer is configured as an Encoder, assuming the inputs to the transformer ...
sudar's user avatar
  • 1
0 votes

How are sentences turned into a vector in LLM

Aggregating embeddings from pretrained models If you want a single vector representation from a vanilla language model (i.e., one not specially trained for producing sentence embeddings) like GPT you'...
Alexander Wan's user avatar
1 vote

Why do Transformer decoders use masked self attention when producing new tokens?

Its all about speed. During training: you use teacher forcing and you feed the entire target sequence $Y$ to the decoder (say of length $N$). You want the decoder to attend to ...
pi-tau's user avatar
  • 702
0 votes

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

TL;DR YES. If the sequence length of $Q, K, V$ is $L$, the embedding size is $E$, and the number of heads is $H$, then weight matrices are of the order $E \times (E // H)$ to transform $E$-sized ...
sudar's user avatar
  • 1
0 votes

What is the intuition behind self-attention?

One way to think about this is that self-attention does a summary of what is going on, either in text or images. But rather than creating a new representation for the summary, the summary is ...
drewlr's user avatar
  • 1
1 vote

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?

They all have different weight initializations. I think that the chance of gradient descent discovering the same local minima in such high dimensions is low.
JobHunter69's user avatar
1 vote

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

During the training, we would process the entire sequence at once and train the Transformer with the teacher-forcing algorithm usually. If the input sequences vary in length, we would truncate or pad ...
entropy07's user avatar
0 votes

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

Yes, the query, key and value are the entire sequence. If the sequence is smaller than the maximum sequence length (which is the size of the linear layers), then you use padding tokens to complete the ...
c p's user avatar
  • 111
1 vote

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I think that the idea of "reduced effective resolution" from averaging is best understood through seeing how the proposed multi-head attention architecture fixes the issue. Specifically, ...
Andrew Du's user avatar
1 vote

Is the multi-headed projection matrix in self-attention redundant?

The technical reason is the residual connection around the self-attention block (first line in your code: x += self_attention(x)). The transformation $W_O$ is ...
Chillston's user avatar
  • 1,674
1 vote
Accepted

Attention with Recurrent Neural Networks

You don't need padding for attention with variable-length inputs. Looking at the formulation in the article: A learned model ($\mathbf{a}(\cdot)$) encodes the hidden state of each input token ...
Alexander Wan's user avatar
0 votes

Attention with Recurrent Neural Networks

Two concepts are critical here: masking and padding. From Tensorflow: Masking is a way to tell sequence-processing layers that certain timesteps in an input are missing, and thus should be skipped ...
Brian O'Donnell's user avatar

Top 50 recent answers are included