Are transformer models better than comparable-complexity MLP-based models?

Question

I've watched the outstanding Andrej Karpathy's From Zero to Hero course. In the last lecture, he introduces Transformer decoder architecture, which is able to produce Shakespear-like text. However, there was no direct comparison of the achieved cross-entropy loss (~1.4) with simple MLP models he talked about in the first 5 lectures.

What if one trains an MLP based model with a similar number of parameters/layers, the same context length and also including layer normalization, feed forward and dropout, would the result be substantially worse? Would the training take longer? Are there direct comparisons like that in the literature?

Answer 1

I can't provide you with numbers and results, but I'd expect (for not triavial problems) the MLP-based model to be worse than a Transformer.

The reason is that transformers are designed to handle sequences, whereas MLP are not (even if you flatten the entire sequence into a vector). Trasformers can leverage an inductive bias specifically designed for sequences (likewise CNNs for images), which should account for a better learning, at least, even in case of similar number of parameters and regularization.

Thanks to self-attention transformers are able to learn to focus on relevant parts of the sequence, even far in time, whereas simple MLPs cannot do this. Moreover, positional encoding provides the transformer to use the sequence order information, enabling it to learn to attend even to relative positions. Instead, MLPs just combine information from all the vectors' components.