Wrong with nothing missing

What does it mean for data to be wrong when every field is filled in and every rule passes?

A dashboard shows a green tile: data quality 99.8 percent. Every not-null constraint holds, every date parses, every foreign key resolves, no duplicates. And the number on the report beside it is nonetheless wrong, and someone has made a decision on it.

That combination is the thing worth sitting with. If every check passes and the data is still wrong, then the checks were never testing what we thought they were testing. What exactly were they testing?

Take a single field: a customer's date of birth, recorded as 1970-01-01.

Is it populated? Yes. Does it parse as a date? Yes. Is it in a plausible range — not in the future, not implying an age of two hundred? Yes. Every rule passes. And yet if that customer was born in 1984, the value is simply false, and no amount of checking the shape of the value will ever reveal it.

Now notice why. Every one of those checks is a comparison between the data and a rule we also wrote. The date-of-birth question is a comparison between the data and the world. Those are categorically different operations, and only the first can be done inside the database.

That is the fault line running through the whole subject. The widely used six dimensions — completeness, validity, uniqueness, consistency, timeliness, accuracy, as set out by the DAMA UK working group in 2013 — do not all sit on the same side of it. Completeness, validity, uniqueness and consistency are all internal: they can be evaluated with nothing but the data and the rules. Accuracy and timeliness are external: they require a referent outside the system — reality, or a clock against a business expectation.

Follow the consequence through. What does an organisation measure? The internal ones, because they are cheap, continuous, and automatable. What does it report? A quality score. And what does that score then get read as? A statement about accuracy — which is the one thing it never measured. The green tile is real evidence, but it is proxy evidence, and the substitution happens silently.

Then a further turn, and this is the one that makes the topic genuinely hard. Suppose we did have a referent, and every value matched reality perfectly. Is the data then good?

Consider a customer table that is flawlessly accurate as of the moment each record was captured — and that captures address at first order and never again. For a mailing campaign that is unusable; for reconstructing what happened at the time of an order it is exactly right. Same data, same accuracy, opposite verdicts. Which tells us quality is not a property of a dataset at all. It is a relation between a dataset and a use. That is what fitness for purpose means, and it is why a quality programme that never asks what the data is for produces scores nobody can act on.

There is one more failure that passes every check: data that is individually correct and collectively meaningless. Two systems each holding an internally consistent definition of "active customer" — one counting anyone who has not cancelled, the other anyone who has ordered in ninety days — will both pass every rule and disagree by thirty percent, and the disagreement is not a data error. It is a definitional one, living in the metadata rather than in the rows, which is why documented definitions are a quality control and not merely paperwork.

The honest limit: there is no way to establish accuracy without touching the world. Sampling and manual verification, comparison against an authoritative external source, or a feedback signal such as returned post — these are the only routes, and all of them cost money and none of them are continuous. That expense is precisely why the proxies became the practice.

Three refinements.

First, the misconception to retire: validity is not accuracy. Validity asks whether a value conforms to its rule; accuracy asks whether it corresponds to a fact. A postcode can be perfectly valid and belong to a different house. Every automated quality suite measures the first and is quoted as though it measured the second.

Second, this does not make the internal checks worthless. They are cheap, they run continuously, and they catch the large class of failures that arrive as breakage — a truncated load, a changed source format, a duplicated batch. What they need is honest labelling: they are a smoke detector, not a survey.

Third, and most useful in practice: pick the dimensions from the use. A regulatory report needs completeness and accuracy above all; an operational alert needs timeliness far more than either; a training dataset needs representativeness, which is not on the classic list at all. A quality programme that measures the same six things for every dataset is measuring what is convenient rather than what matters.

• Sampling strategies for estimating accuracy without verifying every record.
• Data observability: catching pipeline breakage as distinct from catching wrong values.
• Why conflicting business definitions produce disagreements no quality rule can find.

Every term the collection defines is gathered in the glossary.