There is no point in picking one of the growing number of articles that come up in a web search for, "Deep learning attention networks," however the bold claims in Attention Is All You Need, Ashish Vaswani et all, 2017 caught my attention earlier. Are their claims that attention networks will supercede RNNs and LSTMs credible?


I think there are two parts to answering this question.

First, about the specific paper that has been mentioned.

The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more modest claim that the new attention-based architecture can outperform existing modes on a machine translation task (which is supported with experiments), and that, maybe, in the future, attention-based models could work well in other domains, or provide advantages in interpretability.

However, there's also a second way to interpret this question. Attention-based networks are getting a lot of buzz in the AI community this year and to some extent last year. It seems likely that there are a lot of research efforts underway to apply them to other kinds of problems. If these efforts pan out, then they could indeed become the tool of choice.

At this point however, the answer to whether they will has to be "Maybe". There is no good underlying theory for why some NN architectures work better than others on a given kind of problem. We have post-hoc explanations for some (like, for instance, CNNs), but in the absence of a general theory, we're better off waiting for the empirical results before making a call. In particular, the No Free Lunch Theorem tells us that if Attention-based networks work very well on one kind of problem, we should actually expect them to work poorly on others! I suspect we'll have a better answer to this question in 1-2 more years however.

  • $\begingroup$ and.. it almost 2 years from your answer, what do you think :) $\endgroup$ – malioboro Jul 9 '20 at 14:36
  • 3
    $\begingroup$ @malioboro The data are in, and attention-based networks are the new state-of-the-art on a very wide range of language tasks, so the answer is yes, at least in language, and at least for now! $\endgroup$ – John Doucette Jul 9 '20 at 21:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.