0
$\begingroup$

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the output.

Can an existing transformer model, such as GPT-2, be modified to perform the same task on a sequence of numbers and estimate the next most probable number? If so, what modifications do we need to perform (do we still train a tokenizer to tokenize integers/floats into token IDs?)?

$\endgroup$
0
$\begingroup$

To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:

  • the odd numbers
  • the primes
  • numbers expressed in digits, but listed in alphabetical order of their name in German
  • numbers listed in the lexical order of the reverse of their representation in base 3
  • the phone numbers in the Manhattan phone director, listed in alphabetical order of subscriber
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.