There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + Attention, as well as for the Transformer. Assume as many things about the architecture as possible are held equal between the two models. Would the inference of the RNN + Attention still be worse than that of the Transformer?

This post hints at yes; in the original "Attention Is All You Need" paper, the Transformer outperformed not only in cost but also in BLEU. It seems like they experimented to a degree where one could say it's not the optimization process / BPTT that failed in the RNN + Attention models, but truly when the model was saturated the performance still was worse.

Why is this?

Intuitively I feel like the set of programs that can be expressed as "stateful" programs is a strict superset of the set of programs that can be expressed as "stateless" programs, such that the only disadvantage would be that it's harder to find the optimal "stateful" program.



You must log in to answer this question.