13 Jul 2024 AI Engineering5 min read

Structured Document Extraction with Vision LLMs

In 2024, vision LLMs can turn invoices and contracts directly into schema-validated JSON. We examine what GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro deliver on documents, which accuracy ranges are realistic in production, why the validator comes before the prompt, and where human review remains indispensable.

Invoices arrive as pixels

An invoice is not text. It is a layout: a supplier block in the top left, an amount table in the middle, payment terms in a footnote. Plain PDF text extraction discards this geometry. Scanned pages and smartphone photos contain no text layer at all. Contracts add a second problem: the clause that matters sits somewhere in thirty pages of boilerplate.

The traditional answer is OCR plus templates — per-supplier rules that map page coordinates to fields. Templates work until a supplier changes its layout, and maintaining hundreds of them creates a permanent operational burden. The more valuable target is therefore different: any document in, schema-validated JSON out, even for layouts the system has never seen.

The task can be stated precisely: from a document image and a target schema, produce a JSON object that conforms to the schema, contains only values evidenced by the document, and marks every other field as null. This definition is not a formality; every design decision in the article follows from it.

A document arrives — scan, mail or upload. 1/4

Vision models read the page directly

In 2024, this approach became practical. GPT-4o (13 May 2024), the updated Gemini 1.5 Pro (generally available since May 2024), and Claude 3.5 Sonnet (20 June 2024) all accept page images directly. That removes the separate OCR step: the model sees tables, stamps, checkboxes, and handwritten margins as they appear to a human clerk.

DocVQA (Mathew et al., 2021) is the standard benchmark for document understanding: 50,000 questions over 12,767 document images, scored by ANLS. The figures below are those reported in Anthropic's model card addendum of June 2024. One caveat: DocVQA asks single questions. Extracting a complete schema with dozens of interdependent fields is a harder task, and no benchmark number transfers to it one to one.

Model	Release	DocVQA (test, ANLS)
Claude 3.5 Sonnet	Jun 2024	95.2%
Gemini 1.5 Pro	May 2024 (GA)	93.1%
GPT-4o	May 2024	92.8%
GPT-4 Turbo	Apr 2024	87.2%

Schema first not prompt first

We recommend a fixed order: schema first, prompt second. The target schema is JSON Schema — types, enums, format constraints, required fields. A field the document does not contain must be null, never a guess; that rule goes into the schema description and into the prompt. The schema is also the contract the downstream system tests against, which makes it the natural place for domain knowledge such as allowed currency codes.

The 2024 APIs help but do not close the loop. OpenAI's JSON mode (November 2023) guarantees syntactically valid JSON — not conformance to your schema. Function calling (June 2023) and Anthropic's tool use (generally available since the end of May 2024) accept a schema and bias the output strongly toward it. None of them guarantees conformance. Every response must pass a validator; that is not paranoia but the design.

Accuracy in benchmarks and in production

Moving from benchmark to production means a distribution shift. Real documents are skewed scans, faxes, and photographed paper; real schemas contain 20 to 50 fields per document rather than a single question. We therefore state our project observations deliberately as ranges: header fields of digitally created invoices — supplier, date, gross amount — land above 95 percent; line items, handwriting, and stamps score noticeably lower, with degraded scans lower again.

The dominant error class is long identifiers: IBANs, invoice numbers, order references. A single transposed character is invisible in fluent output. Vision LLMs also fail differently from OCR. They do not produce garbled characters — they produce plausible wrong values, and occasionally a value for a field the document never mentions. A plausible error is worse than an obvious one, because it reads as correct.

Validation before trust

We build these pipelines validation-first: the validator exists before the first prompt is written. Three layers. Syntactic — the output parses and conforms to the JSON Schema. Semantic — IBAN check digits per ISO 13616, VAT ID patterns, parseable dates, known currency codes. Arithmetic — net plus tax equals gross, line items sum to the invoice total, contract dates are ordered.

When validation fails, we allow exactly one retry and include the validator's error message in context. A second failure routes the document to a human. The limit remains clear: validation catches inconsistent values, not necessarily incorrect ones. A misread delivery date can pass every rule. Validation narrows the undetected error space, but does not close it.

Human review where confidence is low

Vision LLMs return no calibrated per-field confidence. OpenAI's GPT-4 technical report (March 2023) showed calibration degrading after RLHF, and token log probabilities are at best a weak proxy at field level. Specialised services — Amazon Textract, Azure AI Document Intelligence — do return per-field confidence scores. That remains a real argument for hybrid pipelines.

A workable substitute is agreement. Extract every document twice — two runs or two different models — and compare per field. Agreement plus passed validation: auto-accept. Disagreement, a validation failure or a business-critical field such as the payee IBAN: a review queue, where a person sees document and extracted value side by side. The goal is that reviewers handle flagged fields, not every document. What fraction gets flagged depends on document quality — honest projects measure it instead of promising it.

Outlook from July 2024

Three expectations from where we stand in July 2024. First, schema conformance moves server-side. Constrained decoding is proven in open source — llama.cpp grammars, the Outlines library — and we expect hard schema guarantees from the large API providers, replacing retry loops on the syntactic layer. The semantic and arithmetic layers remain our job.

Second, price per page will keep falling. GPT-4o already costs half of GPT-4 Turbo in the API, while Gemini 1.5 Flash and Claude 3 Haiku mark the small-model trajectory. Two-model agreement checks should therefore move from luxury to default. Third, human review will not disappear; it will move up the value chain, from typing values off paper to resolving flagged disagreements. We design systems for that role, not for its elimination.