Confident model errors

Why does a language model deliver an invented fact in exactly the tone it uses for a true one?

Ask a model for a real citation and you get an author, a year, a journal, a page range. Ask for one that does not exist and you get an author, a year, a journal, a page range. There is no tremor, no hedge, no shift in register at the boundary between what the model has and what it does not.

The natural reaction is that it should know — if it is uncertain, why not sound uncertain? But that smuggles in an assumption worth exposing: that hesitancy is caused by uncertainty. In a person that is roughly true, because fluency is limited by access to the thing being described. Is it true of a text generator? What, mechanically, would make the sentence come out differently?

Start with what the model produces. At each step it emits a distribution over possible next tokens, and that distribution has a shape — sometimes concentrated, sometimes spread. That shape is often called confidence, and it is a real quantity. But notice what it is confidence about: which continuation is likely under the distribution the model learned. It is not a measurement of the world, and no channel exists by which the world could have entered it at generation time.

So separate two things we habitually merge. The answer and the confidence attached to it are both outputs of the same process; the confidence is not independent evidence about the answer, and it can be wrong in exactly the way the answer can be wrong.

That gives us the tractable question. Not "is the model confident?" but "when it says eighty percent, is it right eighty percent of the time?" That property is calibration: a calibrated system's stated confidences match observed frequencies. Calibration is orthogonal to accuracy — a forecaster who says forty percent chance of rain and is right forty percent of the time is perfectly calibrated and not very accurate, which is still useful. The dangerous combination is the reverse: high stated confidence, lower observed accuracy.

Which is, empirically, where modern neural networks sit. Guo and colleagues showed in 2017 that as networks got deeper and more accurate they also got systematically worse calibrated than the smaller networks preceding them — reliably overconfident, with the gap widening at the high end. Accuracy and calibration were moving in opposite directions while everyone watched only one of them. In-distribution the problem is partly fixable and cheaply so: temperature scaling, a single parameter flattening the output distribution after training, recovers much of the calibration.

Now push on "in-distribution", because it is doing heavy lifting. Ovadia and colleagues found in 2019 that calibration degrades badly under distribution shift, and — more awkward — that methods looking well calibrated on held-out test data do not keep their advantage once the data moves. The fix works precisely where you least need it.

Why should that be? The useful distinction is between two kinds of uncertainty. Aleatoric uncertainty is irreducible noise in the world: a fair coin, an ambiguous photograph, a question with two defensible answers. More data does not remove it. Epistemic uncertainty is the model's own ignorance — about parameters, about regions of input space it has barely seen — and more data does reduce it. A distribution over outputs can express the first; it has no natural way to express the second. Nothing in a softmax says "I have never seen anything like this before", so an input from nowhere still yields a confident-looking distribution.

Apply that to a fabricated citation and the puzzle dissolves. Fluent, authoritative academic prose is what the training data looks like; producing it is the default register, not a signal earned by checking something. There is no lookup step that succeeded or failed, so there is no failure for the tone to reflect.

Three refinements hold this together.

First, "confidence" names two quantities that are easy to conflate: the token-level probability the model assigns, and the verbalised confidence — the model writing "I am fairly sure". These are produced differently and can disagree, so a hedged sentence is not a readout of the internal distribution, and a fluent one is not evidence the distribution was sharp.

Second, training for helpfulness can move calibration. OpenAI reported in the GPT-4 system card that the pre-trained model was well calibrated on a multiple-choice benchmark and noticeably less so after post-training with human feedback — a plausible consequence of optimising for responses people rate well, since people rate confident answers well. Calibration is a property that can be trained away without anyone intending it.

Third, an honest boundary. Calibration is one lens on confident errors, not the whole account — fabrication also has causes in what the training data contained, in objectives that reward guessing over abstaining, and in whether retrieval is wired in at all. And the field is active rather than settled: ensembles, self-consistency, conformal prediction and abstention training all improve matters, none solves it, and all degrade under the same distribution shift that broke the simpler fixes.

The practical consequence is blunt. Since fluency belongs to the generator and confidence is another model output rather than evidence about the world, an answer's tone carries essentially no information about whether it is true — and the only reliable check is external.

• How conformal prediction converts a model's scores into sets with a guaranteed coverage rate.
• Why an ensemble of independently trained models captures epistemic uncertainty a single model cannot.

Every term the collection defines is gathered in the glossary.