Timeline for How is BERT different from the original transformer architecture?
Current License: CC BY-SA 4.0
15 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S May 31 at 2:46 | history | suggested | rfabbri | CC BY-SA 4.0 |
relevant typo : tranformed -> transformer
|
| May 31 at 1:39 | review | Suggested edits | |||
| S May 31 at 2:46 | |||||
| Mar 21, 2022 at 16:31 | comment | added | profPlum | @nbro Perhaps, but IIRC the point-wise FFNs 'slide' across the time-dimension (paperswithcode.com/method/position-wise-feed-forward-layer), which is perhaps a meaningful distinction. But ya, I can't blame you. ...The paper itself could stand to be much clearer on the matter haha | |
| Mar 19, 2022 at 0:01 | comment | added | nbro | @profPlum Thanks for sharing this info. As you can imagine, I gave this answer already a long time ago, so I forgot the details of the paper. However, note that you could also view a "normal" fully connected layer as a convolution layer with a kernel that has the same dimensions as the input. See also this. | |
| Mar 18, 2022 at 23:35 | comment | added | profPlum | @nbro There are convolutions, though the authors call them "Point-wise feed-forward networks" and they discard the traditional conventions of CNNs (hence your quote). BUT the original paper also said this: "Another way of describing this [point-wise feedforward network] is as two convolutions with kernel size 1." | |
| Mar 14, 2022 at 8:54 | comment | added | nbro | @profPlum There are no convolutions in the transformer. The original paper even says "Since our model contains no recurrence and no convolution...". | |
| Mar 13, 2022 at 23:57 | comment | added | profPlum | You said it was supposed to solve seq2seq tasks without convolution, but it does use convolution doesn't it? Right after the attention mechanism dense layers are convolved across the attention outputs + positional embeddings, albeit not with the traditional conventions of CNNs (e.g. there is no flexible stride). | |
| Jul 25, 2021 at 22:00 | comment | added | avocado | Well explained! | |
| Nov 27, 2020 at 6:02 | vote | accept | chessprogrammer | ||
| Sep 21, 2020 at 21:29 | history | edited | nbro | CC BY-SA 4.0 |
added 70 characters in body
|
| Sep 21, 2020 at 16:44 | history | edited | nbro | CC BY-SA 4.0 |
added 18 characters in body
|
| Sep 21, 2020 at 16:36 | history | edited | nbro | CC BY-SA 4.0 |
added 466 characters in body
|
| Sep 21, 2020 at 16:27 | history | edited | nbro | CC BY-SA 4.0 |
added 84 characters in body
|
| Sep 21, 2020 at 16:10 | history | edited | nbro | CC BY-SA 4.0 |
added 31 characters in body
|
| Sep 21, 2020 at 16:04 | history | answered | nbro | CC BY-SA 4.0 |