What Happens When the Model Reads It Wrong: Building LLM Compliance Review That Survives an Audit

AML work teaches you early that the two kinds of compliance mistakes don't cost the same. A false alert costs an analyst an afternoon, while a missed violation can end up in a regulator's findings. I learned that while writing regulatory modules for a banking client in 2014, and it was still on my mind years later at Capital One, when my teams built AI-enabled OCR and LLM workflows for compliance review. Those workflows reduced manual review effort by about 60%. Most of our engineering time went into the cases where the model misread a document and what the system should do next.Twenty Years of Reading Rules for a Living I didn't come to this through machine learning. My first decade was integration and data work: messaging systems, a payment-messaging gateway migration at a bank in Melbourne, and later AML scenario logic built on top of enterprise data marts. Years of that work showed me how many processes in a bank depend on someone reading a document and checking it against a rule. At Capital One, I spent several years scaling Auto Navigator, a customer-facing lending platform, from a pilot to more than a million monthly users. Between 2021 and 2023, I led engineering teams modernizing auto-lending and compliance platforms, and that's where the OCR and LLM work happened.Along the way, I was lead inventor on a patent (US10867268B1) for applying AI to external regulatory and enforcement information to spot emerging compliance risk. I was used to machines reading regulatory text; letting one make the call on a document was new.Why Compliance Review Is a Hard Place for an LLM The error asymmetry is the first problem, and it breaks the metric most teams reach for. A system can be 97% accurate overall and still miss the violations you care about, because the aggregate number blends easy passes with the rare failures that matter. Precision and recall per rule, with recall on violations treated as the number that can block a release, tell you whether the system is safe to ship. The second problem is that every decision has to be explainable after the fact. An auditor can pick any cleared document from last year and ask why it was cleared. "The model was confident" won't satisfy an examiner. You need to trace each decision to the exact text it relied on and the system version that made it.There's also a calendar problem that engineers tend to underestimate. In a regulated company, customer documents can't go to an arbitrary endpoint, so model hosting, logging, and data handling all go through security and compliance review before anyone tests on real data. In my experience, that review can take longer than building the prototype, and plans that leave no room for it slip.The Prototype Everyone Builds First It looks good in a demo. You pick a dozen clean documents, the model returns sensible verdicts with nicely written reasoning, and the room gets excited. It also hides almost everything that will go wrong in production, which is why any pipeline like this should run in shadow mode beside human reviewers before it decides anything. Where That Prototype Breaks Extraction Loss: Scanned documents carry stamps, signatures, skewed pages, faded print, and multi-column layouts. OCR on those pages doesn't fail loudly. It drops a line, merges two columns, or misreads a number, and passes along text that is mostly right. The model has no way to know a clause is missing, so it reasons carefully over an incomplete document. The mistake is treating OCR output as ground truth when nothing downstream carries a signal about extraction quality. Rationales You Can't Check: Ask a broad question and accept free-text reasoning, and some of what comes back will cite clauses that aren't in the document, or paraphrase loosely enough that a reviewer can't find the source sentence. The verdict may even be right. For an audit trail that doesn't help, because nobody can show where it came from. The Long Tail: Test sets get built from the most common templates, since that's what's easy to collect. Production also contains older formats and rare variants that appear a few times a month. Quality drops on those, and nothing in the naive pipeline tells you the model is looking at a layout it was never evaluated on. A stronger model fixes none of this. It can't recover text OCR never produced, and better writing doesn't make a rationale verifiable.The Architecture What follows is the design I'd use today, drawing on that work. It starts from one question asked about every component: when this step fails, where does the document go? Gate Extraction per Page Score OCR confidence page by page and act on the lowest page; one bad page in a clean ten-page document barely moves the average. If any page falls below the floor, the document goes to a reviewer before the LLM sees it. Calibrate the floor against labeled examples rather than picking a round number. I'd add a layout fingerprint too, built from structural features such as page count, section headers, and the positions of key fields. A document that doesn't match any template in the evaluation set goes to review automatically. That one check covers much of the long-tail problem without touching the model.One Rule per Call, With "Can't Tell" as a Valid Answer Replace "is this document compliant?" with one narrow check per rule, each returning structured output that gets validated before anything else happens: from dataclasses import dataclass from typing import Literal, Optional Verdict = Literal["pass", "fail", "insufficient_evidence"] MAX_RATIONALE = 500&nbsp; # characters @dataclass class RuleFinding: &nbsp;&nbsp;&nbsp;&nbsp;rule_id: str &nbsp;&nbsp;&nbsp;&nbsp;verdict: Verdict &nbsp;&nbsp;&nbsp;&nbsp;evidence_quote: Optional[str] &nbsp; # verbatim text from the document &nbsp;&nbsp;&nbsp;&nbsp;page: Optional[int] &nbsp;&nbsp;&nbsp;&nbsp;rationale: str&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # short; capped in the prompt and validated below def parse_finding(raw: dict) -> RuleFinding: &nbsp;&nbsp;&nbsp;&nbsp;verdict = raw.get("verdict") &nbsp;&nbsp;&nbsp;&nbsp;if verdict not in ("pass", "fail", "insufficient_evidence"): &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise ValueError(f"invalid verdict for {raw.get('rule_id')}") &nbsp;&nbsp;&nbsp;&nbsp;quote = raw.get("evidence_quote") &nbsp;&nbsp;&nbsp;&nbsp;if verdict != "insufficient_evidence" and not quote: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise ValueError("pass/fail requires an evidence quote") &nbsp;&nbsp;&nbsp;&nbsp;rationale = raw.get("rationale", "") &nbsp;&nbsp;&nbsp;&nbsp;if len(rationale) > MAX_RATIONALE: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;raise ValueError("rationale too long") &nbsp;&nbsp;&nbsp;&nbsp;return RuleFinding( &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rule_id=raw["rule_id"], &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;verdict=verdict, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;evidence_quote=quote, &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;page=raw.get("page"), &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rationale=rationale, &nbsp;&nbsp;&nbsp;&nbsp;) insufficient_evidence is the verdict I'd fight hardest to keep. Without it, the model must choose pass or fail even when the document doesn't contain what the rule needs, so it guesses, and its guesses read as confidently as its correct answers. Adding it lets "this document doesn't let me decide" become a valid output that routes to a person.Scoping by rule also makes debugging tolerable, since a misbehaving check traces back to a single rule and its prompt. The cost is more calls per document. Sending each rule only the section it applies to, running checks in parallel, and giving simple field-presence rules to a smaller model brings that back under control.Verify That the Evidence Exists Every pass or fail must quote the document, and the quote gets checked against the extracted text before the finding counts: from difflib import SequenceMatcher def normalize(text: str) -> str: &nbsp;&nbsp;&nbsp;&nbsp;return " ".join(text.lower().split()) def evidence_found(quote: str, page_text: str, min_ratio: float = 0.9) -> bool: &nbsp;&nbsp;&nbsp;&nbsp;q, p = normalize(quote), normalize(page_text) &nbsp;&nbsp;&nbsp;&nbsp;if not q: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False &nbsp;&nbsp;&nbsp;&nbsp;if q in p: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return True &nbsp;&nbsp;&nbsp;&nbsp;# find the best-aligned window, then score it, so small OCR misreads &nbsp;&nbsp;&nbsp;&nbsp;# inside a real quote pass while paraphrases fail &nbsp;&nbsp;&nbsp;&nbsp;m = SequenceMatcher(None, q, p, autojunk=False).find_longest_match(0, len(q), 0, len(p)) &nbsp;&nbsp;&nbsp;&nbsp;start = max(0, m.b - m.a) &nbsp;&nbsp;&nbsp;&nbsp;window = p[start:start + len(q)] &nbsp;&nbsp;&nbsp;&nbsp;return SequenceMatcher(None, q, window, autojunk=False).ratio() >= min_ratio Exact matching is too strict, because a misread "1" and "l" rejects a correct quote. A loose threshold lets paraphrase through, and then the check proves nothing. Tune the ratio against labeled examples of real quotes and of paraphrases, and expect it to take a few rounds.Rules that check for the absence of something need a different evidence shape, such as a reference to the section that was searched, since a correct pass on those has nothing to quote.Route by What the Error Costs The system finalizes one kind of outcome on its own: a clean pass on a lower-risk document. Every other outcome goes to a person. def route(doc, findings, verified) -> tuple[str, str]: &nbsp;&nbsp;&nbsp;&nbsp;if doc.min_page_ocr_conf < OCR_FLOOR: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "low_ocr_confidence" &nbsp;&nbsp;&nbsp;&nbsp;if not doc.template_known: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "unknown_template" &nbsp;&nbsp;&nbsp;&nbsp;if not all(verified): &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "evidence_not_found" &nbsp;&nbsp;&nbsp;&nbsp;if any(f.verdict == "insufficient_evidence" for f in findings): &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "insufficient_evidence" &nbsp;&nbsp;&nbsp;&nbsp;if any(f.verdict == "fail" for f in findings): &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "potential_violation" &nbsp; # a person confirms every fail &nbsp;&nbsp;&nbsp;&nbsp;if doc.risk_tier == "high": &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return "human", "high_risk_tier" &nbsp;&nbsp;&nbsp;&nbsp;return "auto_clear_sampled", "clean" The reason codes also work as an operational dashboard. If the low_ocr_confidence share jumps, something changed at intake, maybe a new scanner setting or a new document source, and you can see it long before anyone downstream complains.Reviewers should never get a bare "please check this." Give them the source document with the model's cited evidence highlighted and the reason it was routed. Gabriel le Roux argues that a checkpoint placed after the expensive work, with nothing to help the reviewer, tends to become a rubber stamp.Let Compliance Write the Answer Key The evaluation set should be labeled by compliance professionals. Engineers can build the harness, but the people accountable for the outcome should decide what correct means. Once compliance owns the labels, a release conversation sounds less like "convince us this is safe," and more like "recall on this rule dropped, what changed?" Score each rule separately and make violation recall a hard gate:def release_gate(candidate: dict, baseline: dict, max_drop: float = 0.0): &nbsp;&nbsp;&nbsp;&nbsp;"""candidate/baseline: rule_id -> metrics with .recall_fail""" &nbsp;&nbsp;&nbsp;&nbsp;for rule_id, metrics in candidate.items(): &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if rule_id not in baseline: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False, rule_id &nbsp; # new rules need a baseline before they can auto-clear &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if metrics.recall_fail < baseline[rule_id].recall_fail - max_drop: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False, rule_id &nbsp;&nbsp;&nbsp;&nbsp;missing = set(baseline) - set(candidate) &nbsp;&nbsp;&nbsp;&nbsp;if missing: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return False, sorted(missing)[0] &nbsp; # every baselined rule must be re-evaluated &nbsp;&nbsp;&nbsp;&nbsp;return True, None Run it on every change to a prompt, model, rule definition, or OCR setting. Any rule whose recall can't clear the bar stays with human reviewers until it can.Make Every Decision Replayable { &nbsp;&nbsp;"document_id": "doc-…", &nbsp;&nbsp;"document_hash": "sha256:…", &nbsp;&nbsp;"ocr_engine_version": "…", &nbsp;&nbsp;"min_page_ocr_confidence": 0.0, &nbsp;&nbsp;"template_fingerprint": "…", &nbsp;&nbsp;"rule_set_version": "…", &nbsp;&nbsp;"model_id": "…", &nbsp;&nbsp;"prompt_version": "…", &nbsp;&nbsp;"raw_model_response": "…", &nbsp;&nbsp;"findings": [{ "rule_id": "…", "verdict": "pass", "evidence_quote": "…", "page": 0, "evidence_verified": true }], &nbsp;&nbsp;"route": "auto_clear_sampled", &nbsp;&nbsp;"route_reason": "clean", &nbsp;&nbsp;"reviewer_decision": null, &nbsp;&nbsp;"timestamp": "…" } With the version fields, you can show exactly which configuration produced a year-old decision. Store the raw model response as well, so the audit doesn't depend on the model giving the same answer twice. Write records to append-only storage so nobody, including the system itself, can rewrite history.Audit What Nobody Reviews Auto-cleared documents are where unnoticed errors pile up, so send a fixed percentage of them to blind human review every week. The result is an estimate of the escaped-error rate in production, a number an offline test set can't give you. Expand auto-clearing to a new document category only when that estimate stays under a threshold compliance has agreed to. A short detour, because it surprised me. When I later led model-serving work on Kubernetes, the validation gates, load tests, and staged rollouts we built there felt like the compliance lessons in a different setting. The same two habits carried over: version everything, and watch the share of traffic going down each path. I suspect the order mattered and that the compliance work made the serving platform better, though I can't prove it.Why the Remaining Review Work Stays With People People ask why we stopped at 60%. What's left is what a model shouldn't close alone: possible violations, high-risk documents, anything with weak extraction or evidence that can't be verified. The reviewers' day changed. Less of it went to routine clean documents and more to hard cases, with the evidence already in front of them. Dinesh Kumar Elumalai's account of replacing senior developers with AI agents shows the cost of discovering where people are still needed only after the agents are in production. Paul Krause makes a sharper version of the argument in The Case Against Fully Autonomous AI Agents.Where This Approach Struggles Without domain experts willing to label an evaluation set, none of the per-rule gating works, and you have no defensible way to decide what's safe to automate. I'd stop there, before writing a single prompt. Rule changes are the part I'd plan for most carefully. When a requirement changes mid-quarter, there's a window where the rule text, the prompt, and the evaluation labels disagree with each other. Versioning all three together helps, and so does routing everything that touches the changed rule to people until the labels catch up. I haven't seen a clean solution.And if most of your documents fail the extraction gate, the model isn't your problem. Fix intake first. David Iyanuoluwa Jonathan's RAG is a Data Problem Pretending to Be AI covers the same lesson from the retrieval side.What Decides Whether It Ships Models now read compliance documents well enough that they’re rarely the limiting factor. What decides whether a system like this belongs in a regulated workflow is everything built around the model to make its decisions checkable. If I had to reduce this to a few rules: Gate the input before the model can see it. Let the model say “I can’t tell.” Don’t accept a reason you can’t trace back to the document. Route each decision by what an error would cost. Let the people accountable for the outcome define what correct means. Make every decision replayable a year later. Keep checking the decisions nobody reviews. None of these are specific to compliance. The same questions come up wherever a model reads a document, and someone has to stand behind the result: claims, underwriting, KYC, medical records.I now work on AI for customer servicing at a large insurer, where the question is still the same: what happens when the model reads something wrong? If your team runs AI in a regulated workflow, I’d like to hear how you handle rule changes and what you keep with people. The comments are open.