How does ChatGPT assign confidence level to its answer? What is the mathematical description of it?
Here is my guess. It produces many answers from auto-regression. It already has a probability assigned to each of its outputs. It can just output this probability or some function of this probability for the chosen output. Is this correct?
Uncertainty is usually seen under the lens of entropy (aka chaos).
The higher the entropy, the less predictable the outcome. Indeed, it's a paraphrasis of what you are saying... if the entropy is high, the generations will be very different (you are confused). For example, think about asking an LLM to predict a coin flip... it will assign 0.5 probability of both outcomes, and thus if you generate, you will have "very confused" answers.
Instead, for events that are certain, theoretically the distribution will be deterministic, because there is no entropy (you have close to no uncertainty in which state London is, thus it's low entropy)
Given this, the entropy of a distribution is given by $\int p(x) \log p(x)$, and given that an LLM is nothing more than a function that predict distribution, specifically factorized as $p(x) = \prod p(x_i|x_{j<i})$, you can used a factorized entropy to find the overall entropy
In poor words, at every token generation step, you check the softmax distribution the LLM is predicting, you calculate the entropy, and at the end you sum all of the entropies from the generation... the higher it is, the more uncertain is the LLM about the generation
A proxy of entropy, is called perplexity, and it's a very well known metric in the LLM community, here's a nice blogpost about it
ChatGPT does not output confidence estimates for its responses. You can ask the LLM to verbalize a confidence-value for its response, but this is terribly unreliable.
A better approach is to use OpenAI's API which offers logprobs. Using these, you can compute the probability of the response according to the model (remember LLMs are fundamentally probabilistic models). However this approach is still not particularly reliable -- logprobs tend to vary based on how exactly the response happens to be phrased and do not reflect unknown unknowns (sometimes called epistemic uncertainty in the ML literature). Moreover, other LLM APIs like Anthropic do not offer logprobs.
I've built a tool to more reliably estimate uncertainty in responses from any LLM, including OpenAI or Anthropic models. My approach accounts for both aleatoric and epistemic uncertainty in the model, via a combination of LLM self-reflection, probabilistic prediction, and measuring the semantic consistency between alternative responses the LLM finds plausible.
