1
$\begingroup$

Suppose I have included 3 examples of an idiosyncratic sentence for training by a transformer:

  • Example 1: Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes Asdfogiug too.
  • Example 2: Bsodifhas likes Zsdfoiusdhf and Zsdfoiusdhf likes Bsodifhas too.
  • Example 3: Clkwjehrq likes Zsdfoiusdhf and Zsdfoiusdhf likes Clkwjehrq too.

Now we prompt the above-trained transformer with the following:

Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes ______________


Can our transformer complete the prompt correctly?

How would the attention mechanism know to generate "Dlwkjerhtw" (a word which it has never seen before anywhere else) as the predicted next word in our prompt?

$\endgroup$

2 Answers 2

1
$\begingroup$

To complete the pattern based on the given examples, the last statement should follow the same structure. Each example follows a clear pattern where the second person in the first part of the sentence is mirrored as the first person in the second part:

  1. Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes Asdfogiug too.
  2. Bsodifhas likes Zsdfoiusdhf and Zsdfoiusdhf likes Bsodifhas too.
  3. Clkwjehrq likes Zsdfoiusdhf and Zsdfoiusdhf likes Clkwjehrq too.

Following this structure:

Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes Dlwkjerhtw too.

How the Transformer Model Completes the Prompt Correctly

  1. Understanding Patterns: Transformers are highly effective at recognizing and replicating patterns in data. From the given examples, the model identifies the pattern of names being swapped between the clauses. It knows that the first part of the sentence structure “Person1 likes Person2” must mirror as “Person2 likes Person1” based on the examples.

  2. Role of Tokenization:

    • Breaking Down Text: Tokenization involves breaking down the text into smaller units called tokens. For names or words it has never encountered before, the model uses subword tokenization methods like Byte Pair Encoding (BPE) or WordPiece.
    • Handling Unknown Words: These methods decompose unfamiliar words into smaller, more common subword units. For example, "Dlwkjerhtw" might be tokenized into ["Dlw", "kje", "rht", "w"].
    • Maintaining Context: This allows the model to understand and generate text even with previously unseen names, by focusing on the subwords and their context.
  3. Attention Mechanism:

    • The attention mechanism allows the model to weigh different parts of the input sentence when predicting the next token. It helps the model focus on the relevant part of the sentence to determine the appropriate word to generate next.
    • When the model processes "Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes ___________", it attends to the pattern that the second person in the first clause should appear as the first person in the second clause.
  4. Positional Encoding: Transformers use positional encodings to understand the order of words in a sentence, ensuring that the mirrored relationship between the names in the two parts of the sentence is maintained. So, in simple words, it knows the position of words/sub-words in given input.

  5. Contextual Embeddings: Each token, including subwords, is converted into an embedding that captures its context. This helps the model understand the relationship between the names and predict the appropriate completion.

By leveraging tokenization, the attention mechanism, positional encoding, and contextual embeddings, the transformer model can correctly predict that "Dlwkjerhtw" should complete the sentence:

Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes Dlwkjerhtw too.

Even though "Dlwkjerhtw" is a name it has never seen before, the model generalizes the pattern from the provided examples and uses tokenization to handle the new name effectively.

Edit:

The ability of transformers to handle unknown words, such as "Dlwkjerhtw," lies in how they create and use embeddings.

Simplified version of Creating Input embeddings for Transformers

Embeddings in Transformers

Transformers use embeddings, which are learnable representations of input tokens. Let's break down the process:

  1. Input Word: "Dlwkjerhtw"

  2. Tokenization:

    • The model uses a subword tokenization method (e.g., Byte Pair Encoding or WordPiece) to break down the word into known subwords from its corpus/dictionary.
    • Tokenization Result: ["Dlw", "kje", "rht", "w"]
  3. Converting Input to Numbers:

    • Each subword is mapped to a token ID based on the model's vocabulary.
    • Example Token IDs: [15, 20, 30, 8]
  4. Creating Embeddings:

    • These token IDs are converted into embedding vectors. Suppose the length of each embedding vector $d = 512$.
    • The embedding layer of the transformer learns to represent each input token in a meaningful way.
    • Embeddings Shape: 4 (subwords) x 512 (embedding dimension)
  5. Adding Positional Encoding (PoE):

    • Positional Encoding is added to the embeddings to provide information about the position of each token in the sentence.
    • Positional Encodings Shape: 4 x 512
  6. Final Context Embeddings:

    • The final context embeddings are obtained by summing the embeddings and the positional encodings.
    • $Final Contextual Embeddings = Embeddings + PoE$

By combining the subword embeddings and positional encodings, the model captures both the subword information and their positions within the sentence. This allows the transformer to understand and generate text even with previously unseen words, as it can leverage the subword components and their contextual relationships.

For detailed visuals refer to my post PDF on LinkedIN

$\endgroup$
5
  • $\begingroup$ Thank you. I am a little unclear about how transformers learn the rule-based pattern... The word 'Dlwkjerhtw' appears as a completely new vector unlike any other word embedding vectors right? Now then the key, query and value for 'Dlwkjerhtw' will also be different from anything else the transformer has seen. Then after the final processed vector (q, k, v, positional, skipped connection, softmax, etc) is obtained, the final mapping to the one-hot probability distribution will never have learned how to map this vector to 'Dlwkjerhtw'... how would 'Dlwkjerhtw' turn up as the next predicted word? $\endgroup$
    – James
    Commented Jun 10 at 7:53
  • 1
    $\begingroup$ @James, Two things I would like to add: 1. The model leverages subword embeddings, so even if "Dlwkjerhtw" is new, its components are not. The model can generate the correct sequence of subwords based on the learned patterns. 2. The attention scores guide the model to focus on the correct part of the input when generating the output. $\endgroup$ Commented Jun 10 at 20:28
  • 1
    $\begingroup$ @James Also check "edit" part in the updated answer. $\endgroup$ Commented Jun 10 at 21:06
  • $\begingroup$ Thank you for the updated answer! $\endgroup$
    – James
    Commented Jun 11 at 2:21
  • $\begingroup$ It would be great if you could accept and upvote my answer if it has answered your question. $\endgroup$ Commented Jun 11 at 4:29
0
$\begingroup$

After some testing, I believe the answer to the question is no.

From training the following sentences set:

  • Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes [Asdfogiug too].
  • Bsodifhas likes Zsdfoiusdhf and Zsdfoiusdhf likes [Bsodifhas too].
  • Clkwjehrq likes Zsdfoiusdhf and Zsdfoiusdhf likes [Clkwjehrq too].

the resulting attention mechanism will have solidified itself into, for instance (there are several possible alternatives):

  • If the word at first position is 'Asdfogiug', then always output the completion 'Asdfogiug'.

  • If the word at first position is 'Bsodifhas', then always output the completion 'Bsodifhas'.

  • If the word at first position is 'Clkwjehrq', then always output the completion 'Clkwjehrq'.


Now, when given the surprise untrained prompt:

  • Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes ______________

the transformer simply compares (via query-key dot product) which other words happen to have the closest word embedding vector to 'Dlwkjerhtw'. If the embedding vector which happens to be closest is that for 'Asdfogiug', then the completion will be:

  • Dlwkjerhtw likes Zsdfoiusdhf and Zsdfoiusdhf likes [Asdfogiug too]

This is because the transformer has never been told the following analogical rule:

  • If the word at first position is 'Dlwkjerhtw', then always output the completion 'Dlwkjerhtw'.

therefore the production transformer will simply find the closest word embedding vector which it has already trained on, and output that particular completion as trained.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .