Timeline for How is context length calculated in transformers, and why isn't hardware specification considered?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Sep 20 at 12:06 | vote | accept | Parsa Foroozmand | ||
| Sep 19 at 13:27 | comment | added | Alberto | @ParsaForoozmand the architecture has no limitation on the input length, however they are still ML models, thus trained on a distribution of data (in this case, the max-length of the sentences), thus even though it allows it, it might have very poor performances | |
| Sep 19 at 8:51 | comment | added | Parsa Foroozmand | Thanks, so do you mean that in inference time if my hardware allows, i can put as many as tokens that i want and pass the 128k limit but it will perform poorly or it gives an error ? | |
| Sep 18 at 23:15 | comment | added | talles | Hi @Alberto, I've removed the sentence because it might be indeed misleading, thank you for the comment. My understanding is that if you increase the context length ideally you have to retraining/fine tuning it, to better generalize with such bigger context length. I confess I never tested it myself, so I wonder how poorer it will be. | |
| Sep 18 at 23:12 | history | edited | talles | CC BY-SA 4.0 |
deleted 98 characters in body
|
| Sep 18 at 19:55 | comment | added | Alberto | To increase the context length the model would need to be retrained. have I skipped a lecture on how a transformer works? | |
| Sep 18 at 15:13 | history | answered | talles | CC BY-SA 4.0 |