Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I have been thinking about how learning via text works on humans: we read words, and often we need to read ahead a few words to understand more clearly the ideas that we read before. Most of the time, just reading the next word in a sentence is not enough for clear understanding.
Has anyone tried (and if no, what is your opinion) to modify GPT to predict the next N tokens instead of the next single token? My intuition says that the loss function would decay faster that way.
Sure, I think there have been studies on modifying language models, to predict multiple tokens at once instead of just one. This technique is known as n-gram language modeling, where the model is trained to predict the next n words in a sequence.
The basic idea of this approach is to better capture the dependencies and interactions between multiple words in a sequence, which can improve the model's ability to generate coherent and fluent sentences. However, this technique comes with some challenges, such as increased (or even impossible to handle) computational cost.
Regarding your intuition about the loss function decaying faster with multiple tokens, it is possible that predicting multiple tokens at once could lead to faster convergence during training. However, this would depend on the specific details of the model and training regime. Ultimately, the effectiveness of n-gram based modeling would need to be evaluated empirically on various tasks and datasets.
