Suppose we have a transformer LLM which can do a task such as summarising.

I know transformer can technically handle any input length (assume we are not using learned positional embeddings) because the architecture doesn’t define a fix length for input. However, the quadratic complexity and unavailability of long text data puts a practical limit on sequence lengths used for training and they are not effective beyond the sequence length they are trained on.

Suppose we have enough hardware resources and data of a particular long input length. Will the model be able to perform the same task effectively on that, say, 1 million length? Or will we need a model with more parameters?

I would appreciate any reason or empirical result which shows this would or wouldn’t work. Also will this depend on the complexity of a task? If so why would the said complexity increase with input length?

  • $\begingroup$ Hi @Emoticon and welcome to AI Stack Exchange! If possible, please only ask one question per post. That way, your question is more likely to be answered more quickly. Thank you, and we hope to see more of your posts here soon! $\endgroup$
    – DeepQZero
    Commented Jun 15 at 21:58


You must log in to answer this question.

Browse other questions tagged .