Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.

Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?

  • $\begingroup$ Are you asking whether the architecture of GPT-2 could be trained from scratch to perform the task of predicting the next number or are you asking whether the trained GPT-2 model could perform number prediction zero-shot (without any fine-tuning)? $\endgroup$ Commented Aug 22, 2022 at 7:50

1 Answer 1


To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:

  • the odd numbers
  • the primes
  • numbers expressed in digits, but listed in alphabetical order of their name in German
  • numbers listed in the lexical order of the reverse of their representation in base 3
  • the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber
  • $\begingroup$ Would a en.wikipedia.org/wiki/Mixture_of_experts be an appropriate tool to deal with this scenario? $\endgroup$ Commented Aug 22, 2022 at 8:25
  • 1
    $\begingroup$ No. The set of all sequences of whole numbers is uncountably infinite. Even the set of computable sequences of numbers is large and hard to deal with. Like any other machine learning problem, making progress depends on prior knowledge. Here is a subset of the problem which is hard but not impossible, and which relates to some work by Kolmogorov: a sequence of numbers is output by a program, and short programs are considered more likely than long programs. Given the first N terms, where N is large compared to the expected length of the program, predict the N+1 th term. $\endgroup$ Commented Aug 23, 2022 at 10:47
  • $\begingroup$ The odd numbers are very easy: x(n)=2n+1. If we use x(n)=x(n-1)+2, that includes the even numbers as well. If we use x(n)=2x(n-1)-x(n-2), that is all arithmetic sequences. $\endgroup$ Commented Aug 23, 2022 at 17:38
  • 1
    $\begingroup$ Yes, but this sequence is not easy: 3,5,7. Is that odd numbers>1? Or is it primes>2? Or the start of 3,5,7,3,5,7,3,5,7, ....? Or any other of an infinite number of computable sequences with that prefix? $\endgroup$ Commented Aug 25, 2022 at 7:22
  • $\begingroup$ The simplest model is for odd numbers >1, linear recurrence of order 2. With order 3 you can get 3,5,7,3,5,7... x(n)=x(n-3). No linear model for the primes. An example that makes me crazier is 1,2,... is the simplest interpretation the counting numbers or the powers or 2? Every kid starts by associating that start with counting numbers x(n)=2x(n-1)-x(n-2), but the order of the linear recurrence for the powers of 2 is smaller: x(n)=2x(n-1) $\endgroup$ Commented Aug 25, 2022 at 11:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .