Writing down what you could not determine

Why is a record of the questions you failed to answer worth as much as the answers?

A discovery tool is pointed at an old system and produces a document: forty rules found, twelve interfaces catalogued, nine tables mapped. It reads as a survey of what is there.

But consider what a reader does with a topic that does not appear. They conclude it does not exist. Is that conclusion warranted? Only if the tool looked everywhere and understood everything it found. And it did not — three files would not parse, one subsystem was written in a dialect the extractor does not handle, and two rules were recognised but their purpose could not be worked out. None of that is in the document. So the document is not a survey of the system; it is a survey of the part of the system the tool could digest, presented as though it were the whole.

Start with the ambiguity that sits under every absence. When a finding is missing, exactly one of three things happened: nobody looked there, somebody looked and there was nothing, or somebody looked and could not tell. Those are wildly different states with identical appearances in the output. The first is a gap in coverage, the second is a genuine negative result, the third is a limit of the method.

Ask what it costs to leave them fused. A reader planning a migration allocates effort to what the document mentions. Everything in the first and third categories is invisible, so it receives no effort — and it does not fail politely at planning time, it fails at cutover. The risk has not been reduced by the survey; it has been hidden by it, which is worse than not having surveyed at all, because now there is a document lending it confidence.

So separate them, and notice what separation buys. If the ledger says "these 412 source members were parsed, these 9 failed with this error, these 3 rules were found but their intent is undetermined", then the unknown has a size. Before, the residual risk was unbounded — anything at all might be missing. After, it is a list you can count, assign, and burn down. That is the actual transformation: an unbounded uncertainty becomes a bounded one. It is the same move as an inventory count that records the shelves you could not reach.

There is a second effect that is easy to miss. A recorded failure is a re-runnable one. "Could not determine the trigger for this rule" carries the artefact, the location, and what was tried, so a later pass with a better tool — or a person who happens to know — can close it and see the delta. An unrecorded failure has to be rediscovered from scratch, and typically is not, because nothing signals that it is there to rediscover.

And a third, less comfortable one: the ledger makes the method auditable rather than only its output. If nine files failed to parse and all nine are from one subsystem, that is not nine small problems, it is one shaped one — the extractor does not handle that dialect. You cannot see that pattern from a list of findings. You can only see it from a list of failures, which is why the failures deserve structure rather than a footnote.

Is there a limit to this? Yes, and it should be said plainly. A ledger of known unknowns says nothing about unknown unknowns — the questions nobody thought to ask. Recording that you could not determine a rule's purpose does not help with a rule you never noticed was a rule. Coverage of artefacts can be measured honestly; coverage of questions cannot, and a ledger that implies otherwise is overclaiming.

Three refinements.

First, this is not a confession of failure and should not be written like one. Every method has a boundary; naming it is what distinguishes a survey from a guess. A report with no undetermined items is not more thorough than one with forty, it is more likely to be concealing them — and an experienced reader treats an empty limitations section as a warning sign rather than a clean bill.

Second, an entry is only useful if it is actionable. "Some COBOL did not parse" is nearly worthless. The member name, the failing construct, the error, what was attempted, and whether the item blocks anything downstream are what let someone else pick it up. The unit of the ledger is a question with an address, not a mood.

Third, distinguish the undetermined from the merely uncertain. "This field's meaning is unknown" and "this field probably means settlement date, on the evidence of three call sites" are different claims deserving different treatment. Collapsing confidence into a binary loses the part a reader most needs — whether to verify or to investigate from nothing. Recording the evidence alongside the confidence is what makes the second kind reusable.

• How to express confidence in a recovered fact so a reader knows whether to verify it or re-derive it.
• Whether coverage of artefacts can ever be turned into a defensible statement about coverage of behaviour.
• How a coverage ledger should age, given that a system changes after it is surveyed.

Every term the collection defines is gathered in the glossary.