I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, just reading the next word in a sentence is not enough for clear understanding.

Has anyone tried (and if no, what is your opinion) to modify GPT to predict the next N tokens instead of the next single token? My intuition says that the loss function would decay faster that way.


1 Answer 1


Sure, I think there have been studies on modifying language models, to predict multiple tokens at once instead of just one. This technique is known as n-gram language modeling, where the model is trained to predict the next n words in a sequence.

The basic idea of this approach is to better capture the dependencies and interactions between multiple words in a sequence, which can improve the model's ability to generate coherent and fluent sentences. However, this technique comes with some challenges, such as increased (or even impossible to handle) computational cost.

Regarding your intuition about the loss function decaying faster with multiple tokens, it is possible that predicting multiple tokens at once could lead to faster convergence during training. However, this would depend on the specific details of the model and training regime. Ultimately, the effectiveness of n-gram based modeling would need to be evaluated empirically on various tasks and datasets.



You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .