0
$\begingroup$

In his paper introducing SHA-RNN (https://arxiv.org/pdf/1911.11423.pdf) Stephen Merity states that neglecting one direction of research (in this case LSTMs) over another (transformers) merily because the SOTA in transformers are due to using more computing power is not the way to go.

I agree that finding neat tricks in AI/ML is equally (if not more) important than just throwing more computing power at the problem. However I am a little bit confused.

The main difference (since they both use attention units) between his SHA-RNN and transformers seem to be the fact that SHA-RNN uses LSTMs to "encode the position of words", where transformers do position encoding by using cosine and sine functions.

My confusion comes from the fact that LSTMs need to be handled sequentially and thus they cannot use this large advantage of GPUs, being able to compute things in parallel, whilst transformers can. Wouldn't this mean that (assuming LSTMs and positional encoding are able to acquire the same results), training using LSTMs would take longer than transformers and thus need more computing power, thus defeating the initial puporse of this paper? Or am I misinterpreting this?

Basically my question comes down to "Why would an SHA-RNN be less computationally expensive than a transformer?"

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Transformers can be trained with parallelization. So you might have 100 cores training the transformer for an hour (100 total hours of CPU time), whereas the LSTM has one core training for ten hours (10 total hours of CPU time). Even though the LSTM takes longer, it is less expensive to train.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is what I also had as a final conclusion. So what this means is that if we would train a transformer for 10 hours of CPU time, or a LSTM for 10 hours of CPU time, results might be the same? And that if we invest more time in researching LSTMs we might even find them to be better (performance wise) & less compute expensive? $\endgroup$
    – Tibo Geysen
    Jul 8, 2020 at 12:34
  • $\begingroup$ Right, although to be honest I'm not familiar with the comparison of LSTMs and Transformers in terms of total training time—not sure if they're exactly the same or if one is better $\endgroup$
    – Simon Alford
    Jul 8, 2020 at 13:00

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .