As well as decent modelling of purposes or strings of digits, ChatGPT can identify when a completely novel string might be a product name, a fictional person or place etc.
The language model in ChatGPT is partially character based. There are around 100,000 tokens in latest GPT - the encoding is called
c100k_base. Most of them are common English words (e.g. "the"), but it also supports individual characters including individual digits, and processes sequences of characters and effectively models groups of them at a time.
From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator.
That's a reasonable analogy. ChatGPT is a bit like a 2048-gram prediction engine for the next token, shift the sequence along one and repeat. No different to toy fantasy name generators when viewed from 10,000 feet up.
As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.
A couple of misunderstandings here. First, the random number will not become a single token, but will be one token per digit, or pair of digits or triple digits, depensing on sequence - you can give this a try to help visualise it, in the encoding that ChatGPT uses. Of course each of those tokens will have been seen before, millions of times in the training data.
Second, sequences do not need to be seen in the training data in order for ChatGPT to work with them. In fact, with an input sequence length of 2048, pretty much all inputs to ChatGPT in inference mode are unique never-seen-before sequences. Regardless if some of the tokens represent a long random number, the chances of any 2048 long sequence of letters and short words being unique when generated are very high.
This is where the neural network model differs from a true 2048-gram. It has generalised from the training data well enough that it actually can predict meaningful and useful values for probability of next token, even though in all likelihood it has never before seen the exact same sequence. In this regard it is an approximation of a "perfect" 2048-gram prediction engine that somehow been trained on infinite human writings.
A lot of language modelling is about correctly processing the context of a subsequence, so recognising a number sequence as being a grammatical "unit" that can be reused as-is is not a surprising feature.