We all know that a good translation cannot be done just by splitting words, converting them, and concatenating them back. Otherwise, a dictionary would be just enough. One translation problem is on the alignment of the words.
Uma maçã grande e vermelha
(1) (2) (3) (4) (5)
| \ / _______/
| X /
| / \ /
| / X
| / / \
(1) (3) (5) (2)
A big red apple
This article starts by showing how a RNN translator works and their underlying difficulties. And alignment is a huge pain for RNN because either you'd need another method* to solve it, so RNN could focus on smaller tasks each time.
*And this method usually requires a labeled datasets (like the example above), which is quite tedious to create.
What if, instead of hacking an external element to guess the alignment, we could just send the whole text and train a single neural network to both:
- Somehow solve the alignment problem.
- Use that to predict the next word.
Wouldn't that be awesome? Introducing:
It has a bult-in self-attention component that scores all previous words according to their relevance for the next translated word!
Transformers will automatically solve alignment while translating.