Extracting information from a document always looks easy in a demo. You hand the model a clean invoice, and it pulls out the amount, the date, and the number correctly. But real documents are not like that. Skewed scans, broken OCR, layouts that differ from one organisation to the next, tables that have run together, sometimes handwriting. The difference between a demo system and a reliable one shows up exactly in that βmessy tail.β
This is a general problem; its shape differs across domains β invoices and financial forms, insurance documents, administrative records, reports β but the core is the same: how do we make the model stay stable even on messy input? The answer is not a bigger model. Itβs a better system.
Why reliability is not a property of the model
The common temptation is to think the problem is solved by a stronger model. But reliability isnβt something that lives only in the modelβs weights; it is a property of the whole system β and that means it can be designed and measured.
A model might do fine on a clean document and misread a single figure on the same document with a bad scan. The most dangerous case isnβt that it makes a mistake; itβs that it makes the mistake with full confidence and you donβt notice. In a reliable system, the model is allowed to say βIβm not sureβ β and the system knows what to do with that uncertainty.
Break the problem down, then measure
The first step is to break βdocument extractionβ into specific fields and define, for each one, what βcorrectβ means. A total amount is an exact figure; a date has a defined format; a partyβs name must match the text of the document. Once you have defined correctness field by field, you can finally measure it.
And measuring requires a real evaluation set: examples of the same messy documents you actually face, paired with the correct answer for each field. That set is your ground truth. You measure every change to the system against it: did this change make extraction better, or did it just look better on a few lucky examples?
The numbers we report are evaluation results on our own internal test set, not the outcome of a customer project. That distinction is fundamental to us: a number only means something when you know which set it was measured on and with what definition of βcorrect.β
Turn uncertainty into a feature
The most important design decision is this: what happens when the model is not sure? In a brittle system, uncertainty is hidden and the wrong output quietly moves on. In a reliable system, uncertainty is a signal: the uncertain field is flagged and set aside for human review.
This is what honest AI looks like. The system doesnβt need to be right about everything; it needs to know where it is sure and where it isnβt β and to surface that boundary instead of hiding it. Combining βautomatic extraction on the confident casesβ with βhuman review on the borderline casesβ produces something that is both fast and dependable.
The transferable idea
The core of this goes well beyond documents. Anywhere AI has to work on messy real-world input, the same principle holds: you donβt buy reliability, you build it and measure it. Define correctness precisely, measure it on real examples, and surface uncertainty instead of burying it.
On a clean document, any model is a hero. Trust is earned in the messy tail β and there, you win not with a bigger model, but with a better-measured system.