1
$\begingroup$

I was reading about a study on neural scaling laws from 2017 and they noted this as a summary. From Hestness, Joel; Narang, Sharan; Ardalani, Newsha; Diamos, Gregory; Jun, Heewoo; Kianinejad, Hassan; Patwary, Md Mostofa Ali; Yang, Yang; Zhou, Yanqi (2017-12-01). "Deep Learning Scaling is Predictable, Empirically". arXiv:1712.00409 [cs.LG].

This is an excerpt from wikipedia on neural scaling laws

One of the studies was generative LM with an LSTM. Since then of course we use transformers for this which have beaten every benchmark. Transformers are an architecture change meaning that there should be no change in the power here, so why do they perform better?

Is it a matter of them having a lower loss coefficient here? Is anyone aware of any studies done on the scaling laws between LSTM/Transformers? In the case of a decoder model I would assume the training time for an LSTM is the same given they are both decoding at every step.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Notice the architectural change your reference above is only about optimizer, regularizer, and loss functions which only affect the proportionality factor as claimed, but $D$ here has other architectural contents like the most critical type of learning model employed which would affect the exponent.

And indeed the transformer model including MHA+FFN scales much better than LSTM which would be reflected in the exponent of the neural scaling law, since LSTM training is sequential due to RNN's inherent memory states recurrent nature leading to vanishing gradients and is limited in capturing long-range dependencies. Even a decoder-only transformer at training stage processes the input sequence in parallel to leverage the efficiency of the transformer architecture which has no inherent hidden memory states.

Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM. Therefore from the other dataset size perspective, transformers' number of parameters scale better in the sense of generalization with same training dataset size.

Improve this answer
$\endgroup$
3
  • $\begingroup$ thanks so much for a great answer here, you are right I completely skipped over the details of the architecture specifics they tested over. One thing I am missing from my understanding on your answer is "Additionally since transformers have better domain specific inductive bias, they could achieve lower training loss compared to LSTM." Can you explain this? $\endgroup$
    – Jacob B
    Commented yesterday
  • 1
    $\begingroup$ I mean transformers capture the long-range dependency modeling capacity of language model via self-attention mechanism which is proved empirically (much) better than hidden-sequential-state mechanism of LSTM to capture the same. Transformer's assumed mechanism and model architecture are more useful and accurate for language modeling purpose. Hope this completely clarifies your lingering confusion of this specific question. $\endgroup$
    – cinch
    Commented yesterday
  • $\begingroup$ It does, thanks again @cinch! $\endgroup$
    – Jacob B
    Commented 18 hours ago

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .