Timeline for How does a transformer leverage the GPU to be trained faster than RNNs?
Current License: CC BY-SA 4.0
10 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Mar 19 at 20:01 | comment | added | user77925 | Hey, thanks! I posted the question here: ai.stackexchange.com/questions/45079/… | |
| Mar 13 at 20:56 | comment | added | David | @abcd kind of hard to answer in a comment, probably it would require a writing out of the equations used in attention (perhaps you could ask it in a question and either myself or someone else will answer it), but basically if you look at how self-attention works, you can see that they attend to every token in the sequence without any loss of information (e.g. there is no attempt to collect all prior information in a single dynamic state) | |
| Mar 12 at 17:57 | comment | added | user77925 | I have another question. When you said, This isn't wholly true in transformers, as they attend to every previous token in the context window., can you explain how Transformers are able to do this? I'm kind of new to Transformers. Thanks. | |
| Mar 12 at 17:56 | comment | added | user77925 | Yeah, I think so. Since it's just two LSTMs, it's probably better than standalone LSTM, but still has issues long-term dependencies. | |
| Mar 12 at 16:54 | comment | added | David | I'm less sure on them but I think the same problem would happen. For really long sequences, the hidden state would start to lose information (I think). | |
| Mar 12 at 14:13 | comment | added | user77925 | Thanks, what about bidirectional-LSTM, it's better than normal LSTM and also operate bidirectionally and output contextual embedding same as Transformer ? | |
| Mar 12 at 12:24 | comment | added | David | @abcd kind of, yes. The RNN has access to information from previous time steps through the hidden state, but if you look at how this is updated (in e.g. LSTMs which use a gating mechanism) then it starts to lose more information about time $t$ at time $t+k$ as $k$ increases. This isn't wholly true in transformers, as they attend to every previous token in the context window. | |
| Mar 12 at 0:02 | comment | added | user77925 | Are you saying that RNN/LSTM can only access to information of the previous time step (previous token), while Transformer can access to information of every other tokens in the sequence ? Thanks | |
| Dec 2, 2021 at 21:11 | comment | added | David | please someone let me know if I have messed up the orders of my matrix multiplications. | |
| Dec 2, 2021 at 21:10 | history | answered | David | CC BY-SA 4.0 |