This research note tests whether frontier and open-weight large language models can extract reliable information from UK company accounts, and then audits its own answer key, three times, using the models as the error detector. The models read superbly (97.6–100% after audit, open-weight matching proprietary, memorisation ruled out by a post-release contamination control). The louder finding: every unanimous model "error" turned out to be a fault in the benchmark's own ground truth, including two bugs live in the commercial pipeline. The durable asset is not the model: it is the verification loop.
/1Core thesis
The benchmark asks a commercially important question: can modern LLMs read UK company accounts well enough to extract structured financial facts? The answer is yes: uniformly, across proprietary and open-weight models, including on filings published after every model's release (so provably absent from training data). The durable advantage therefore moves away from model selection and toward data ownership, verification, provenance, and refresh infrastructure.
A revision worth stating plainly: the original version of this note reported that models "invent £0 turnover" for filleted filings. Three rounds of ground-truth auditing falsified that claim: across roughly 970 valid trap trials over five models, no model ever invented a turnover figure. The claim is retired; the full correction history is published with the benchmark.
/2Results
| Model | Clean (n=1,000) | Raw filings, audited (n=331) | Unseen filings (n=348) |
|---|---|---|---|
| GPT-5.5 | 98.4% | 99.7% | 100.0% |
| Claude Opus 4.8 | 99.6% | 99.1% | 100.0% |
| GLM 5.2 (open) | 98.4% | 97.6% | 100.0% |
| Gemini 2.5 Pro | 98.1% | 98.5% | 99.4% |
| DeepSeek V4 Pro (open) | 99.5% | 98.5% | 99.4% |
Differences between models are not statistically significant on raw filings: read as tiers, not rankings. The "unseen" column is the contamination control: filings published 18–30 June 2026, after every model's release.
/3The audit that changed the story
The method: when four or five independent models give the same "wrong" answer, the probability they are all wrong the same way is far lower than the probability the answer key is wrong. Unanimous disagreement is an alarm on the ground truth. Run across three evaluation rounds, that alarm found 26 faults, every one of them in the benchmark, not the models: a sign-handling parser bug, a prior-year fallback that recorded last year's figure when the current year showed "—", trap questions asserting non-disclosure of figures the filings actually printed, and CSS-polluted contexts that buried the target figure. Two of those bugs were live in the commercial data pipeline; the benchmark's most valuable output was debugging the product it was built to justify.
What genuinely remains: a handful of silent, plausible model errors per 350 questions: prior-year column grabs, group-vs-company confusion in consolidated accounts, digit slips (290 read as 390). No warning, no reproducibility. Fine for a demo; disqualifying when a number feeds a lending decision, unless something checks it.
/4Why it matters
The model is no longer the hard part: on single documents the models out-read the reference parser. But their rare errors are silent and non-reproducible, per-figure provenance does not exist at any accuracy, and a full API pass over 3.7 million filings costs five to six figures per refresh against roughly nothing for a pipeline. The winning system is the verification loop: deterministic extraction and model reading cross-checked, disagreements adjudicated against source, corrections dated and published. Each side catches errors the other cannot see.
Plain English takeaway: LLMs make messy filings readable, but reliability at scale comes from verification. The winner owns the cleaned, normalised, provenance-tracked data, and the loop that keeps it honest.
/5Link to the UK Company Financials dataset
This note supports the broader UK Company Financials data product: an ML-ready dataset of UK company filings parsed from Companies House iXBRL accounts, normalised across FRS 102 and FRS 105, with per-figure provenance and the audit discipline described above. The benchmark, evaluation harness, and full correction history are public; the datasets are on Hugging Face.