3
$\begingroup$

What is the reason behind the name "Transformers", for Multi Head Self-Attention-based neural networks from Attention is All You Need?

I have been googling this question for a long time, and nowhere I can find any explanation.

$\endgroup$
7
  • 1
    $\begingroup$ Note that there are other attention-based neural networks that are not "transformers", in the sense that they use recurrent neural networks. In any case, I agree with you, the name "Transformers" is poorly chosen, in that it's not immediately clear why that name was chosen in the first place, so it doesn't provide an immediate intuition about the model. $\endgroup$ – nbro Nov 16 '20 at 12:27
  • $\begingroup$ Right, I specified multi head self-attention and the original paper to make it more precise. $\endgroup$ – Leevo Nov 16 '20 at 12:46
  • $\begingroup$ Maybe the best way to find out why they called it "transformer" is probably to ask the authors of "Attention is all you need" directly. Here are some ideas about why they may have called it "transformer". 1. It's some arbitrary but "nice" name that attracts more people. 2. one of the authors likes the "transformers" movies, 3. one of the authors associated something that transforms with the model. 4. Because of one of the English definitions of "transformer", i.e. "an apparatus for reducing or increasing the voltage of an alternating current.". $\endgroup$ – nbro Nov 16 '20 at 13:56
  • $\begingroup$ Given that the transformer is supposedly more efficient than the equivalent RNNs. See en.wikipedia.org/wiki/Transformer. Maybe it's a combination of some of these options. Only the authors of the transformer can say. So, I suggest that you e-mail them. $\endgroup$ – nbro Nov 16 '20 at 14:04
  • $\begingroup$ I kind of feel guilty for bothering some of these super-scientists with such a noob-ish question. However, I thought about dropping an email too. :) $\endgroup$ – Leevo Nov 16 '20 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.