# What is the purpose of "alignment" in the self-attention mechanism of transformers?

I've been reading about transformers & have been having some difficulty understanding the concept of alignment.

Alignment means matching segments of original text with their corresponding segments of the translation.

Does this mean that, with transformers, we're adding the fully translated sentences as inputs too? What's the purpose of alignment? How exactly do these models figure out how to match the different segments together? I'm pretty sure there's some underlying assumption/knowledge that I'm not fully getting -- but I'm not entirely sure what.

• It's basically: how do the output neurons that are supposed to output "économique" know to connect to the input neurons that see the word "economic"? We could just connect every output position to every input position but then it would have a small maximum size limit and wouldn't be translationally invariant because the weights would record the exact positions (output word 5 -> input word 2). Aug 10, 2021 at 16:37

### Alignment:

We all know that a good translation cannot be done just by splitting words, converting them, and concatenating them back. Otherwise, a dictionary would be just enough. One translation problem is on the alignment of the words. For example:

Uma maçã grande e vermelha
(1)  (2)  (3)  (4)   (5)
|     \ /   _______/
|      X   /
|     / \ /
|    /   X
|    /  / \
(1) (3) (5)  (2)
A  big red apple


### RNN

This article starts by showing how a RNN translator works and their underlying difficulties. And alignment is a huge pain for RNN because either you'd need another method* to solve it, so RNN could focus on smaller tasks each time. *And this method usually requires a labeled datasets (like the example above), which is quite tedious to create.

What if, instead of hacking an external element to guess the alignment, we could just send the whole text and train a single neural network to both:

• Somehow solve the alignment problem.
• Use that to predict the next word.

Wouldn't that be awesome? Introducing:

### Transformers

It has a bult-in self-attention component that scores all previous words according to their relevance for the next translated word!

#### TL;DR:

Transformers will automatically solve alignment while translating.