Although "hallucination" is discussed as if it were a failure mode of token-predicting LLMs (which are the core of all the recent large models), this is not really any different to normal behaviour. LLMs are not designed to output correct information, they are not trained specifically to do so, and there is no loss function or reward signal during training that encourages truth (although some fine tuning or human interaction reinforcement learning applied afterwards may adjust this to some degree).
LLMs do not have inherent filters for detecting true or accurate text output. Instead they output text according to the probability that the model determines that it should appear, based on training material plus the text so far, including any "pre-prompt" setup plus the text supplied by user and it's own output so far.
The extent that LLMs output true and accurate statements is limited to:
The probability that a true statement might appear in all similar text from the training data.
The ability of the model to generalise from the training data and predict accurately.
For some AI systems, ability of added monitoring and control code models to detect and correct issues when they occur. These secondary models are typically far more limited than the LLMs, but may help.
For some LLMs there may be fine tuning that helps the core model provide truer or more useful output. This cannot be done at the same scale as the base training however, because it requires far more human effort per training example.
In all cases, there is limited "understanding" of what the model outputs, in terms of real world relevance and accuracy. Mainly this is achieved because the majority of training data makes logically correct and accurate statements that the model will learn to predict approximately. It cannot refer back to other modes of knowledge though, or have any way to experience the world directly. So sometimes the text predictions produce nonsense that is well written otherwise.
In my experience, this nonsense happens very easily indeed. Just ask a model an expert level question on any subject. The LLM will produce text that on a first look seems reasonable, but an actual expert in the subject will tell you it makes no sense at all in a "not even wrong" kind of way. This is why LLM answers are not accepted on most Stack Exchange sites.
