From what I understand, ChatGPT is just a fancy neural network, operating like a sophisticated Markov Chain generator. As such, it should only be able to generate tokens that are in its training dataset. One thing it should not be able to generate is a token unique to the conversation, such as a random number, since the conversation is not part of its training.
However, ChatGPT very easily, and with complete accuracy, repeats many long random numbers I give it. I also split up the random number across my request, and have ChatGPT piece it together again.
Eric Holloway: How about concatenating these numbers, and removing the intermittent filler words? 9475 3008 3305 11 frog 4357 5075 0021 3385 0894 2324 3812 9336 29 dog 52 1375 8236 3840 5500 6080 3841 log 7266 0722 9112 9774 4967 4707 81
ChatGPT: To concatenate the numbers you have provided, while removing the filler words “frog”, “dog”, and “log”, you can simply combine the numbers in the order that they are given. Here is the resulting number: 9475 3008 3305 1143 5750 7500 2133 8508 9423 2438 1293 3629 5213 7582 3638 4055 0060 8038 4172 6607 2291 1297 7449 6747 0781
How is a neural network able to do this?