Self-consistency

Why does asking the same question five times and taking the most common answer beat asking once carefully?

Here is a procedure that sounds like giving up. Instead of crafting one careful prompt and reading the answer, ask the same question five times with randomness turned on, throw away the reasoning, and keep whichever final answer appeared most often.

It works. On multi-step arithmetic and commonsense reasoning it works substantially better than the single careful attempt, which is the result Wang and colleagues reported in 2022 under the name self-consistency. The puzzle is why. Nothing new was told to the model, no better prompt was written, and four fifths of the work was discarded. So where did the extra accuracy come from?

First we need to see why one attempt is fragile. A multi-step reasoning problem is solved as a chain: set up the quantities, do one operation, carry the result, do another, state the answer. Ask what happens if a single link is wrong. The rest of the chain proceeds faithfully from the error, and the final answer is confidently, fluently wrong. There is no internal alarm, because nothing about producing a wrong intermediate value feels different from producing a right one.

Now ask a second question, about the space of possible chains. For most problems, is there one route to the answer or many? Usually many — you can set up the algebra differently, work in a different order, use a different intermediate quantity. And here is the asymmetry that everything rests on: all correct routes must arrive at the same number, while incorrect routes have nowhere in particular to land. One slip gives 27, a different slip gives 34, a third gives 18.5. Errors scatter; correctness converges.

So imagine sampling forty independent chains rather than one. If, say, half the chains are sound, all twenty of those agree on one value. The other twenty are wrong in twenty different ways and split across a dozen values, none of which collects more than a handful. Which value has the most votes? The correct one — and notice it can win a plurality even though a full half of the attempts were wrong. That is the whole mechanism, and it is genuinely a statistical argument rather than anything about intelligence.

That also explains something that trips people up. Why does self-consistency need the randomness — why not just ask five times at temperature zero? Because deterministic decoding gives you the same chain five times. Sampling is not noise being tolerated; it is the instrument. It is what produces the diverse routes whose agreement carries the information.

And it explains the discarding. The reasoning text is not the output being averaged; the answer is. The technique marginalises over reasoning paths — it asks which final answer is most probable once you sum across all the ways of reaching it — rather than asking which single path is most probable. Those are different quantities, and the second one, which is what a single greedy pass approximates, turns out to be the worse estimator.

Now the honest limits, which matter more than the mechanism because they decide when to use it. Majority voting rescues you from diverse errors. It does nothing whatever about systematic ones. If the model has a consistent misconception — misreads the question the same way each time, or has learned a wrong fact — then all forty chains make the same mistake, agree perfectly, and the vote returns a wrong answer with high apparent confidence. The agreement is not evidence of truth; it is evidence only of convergence, and convergence has two possible causes.

Two more constraints. The answer must be comparable: you need to be able to say two samples gave the same answer, which is easy for a number and awkward for a paragraph, so the technique fits arithmetic, multiple choice and extraction far better than open-ended writing. And the cost is linear in the samples, so the question is always whether the gain is worth that multiple.

One useful side effect: the spread of the votes is itself a signal. Thirty-eight of forty agreeing and fourteen of forty agreeing are very different situations, and that dispersion is a rough confidence estimate you get for free — rough, because a confident consensus can be confidently wrong.

Three refinements.

First, a common misreading: that self-consistency makes the model reason better. It does not touch the reasoning at all. Each individual chain is exactly as good, or as bad, as it was before. What changed is the selection rule applied afterwards — and the gain is entirely a property of aggregating over a distribution rather than sampling one point from it.

Second, the relationship to prompting carefully is additive, not competitive. Self-consistency was introduced on top of chain-of-thought prompting and depends on it: you need the model to produce varied intermediate reasoning for there to be diverse routes to sample. A better prompt raises the accuracy of every individual chain, which raises the fraction of votes that are correct, which makes the majority more reliable. Do both.

Third, the number of samples has diminishing returns. Going from one sample to five buys a great deal; going from twenty to forty buys very little, because once the correct answer is already the plurality winner, more votes only sharpen a decision already made.

• Chain-of-thought prompting, which self-consistency is layered on top of and depends on.
• Weighted voting and verifier models, which score samples rather than merely counting them.
• Ensembling in classical machine learning, and how far the analogy to bagging actually carries.

Every term the collection defines is gathered in the glossary.