Invoice Extraction Accuracy: Source Validation in Python

An invoice extraction returns valid JSON: subtotal 1000.00, tax 190.00, total 1190.00. The amounts add up, but the document actually says 1090.00. Schema validation and arithmetic agreement have both passed while the extraction is wrong.

Accuracy requires comparison with the source. This tutorial adds that layer after structured extraction with Python and Pydantic. It does not replace the model integration or repeat its schema tutorial. It builds an executable post-extraction gate, then explains how to measure errors and review coverage without presenting a successful API response as an accurate invoice.

Separate four checks

Pydantic strict mode can reduce unintended coercions, but it is still type validation. A perfectly typed number may refer to the wrong field. Keep source validation and business decisions outside the claim that a model returned a valid object.

For each extracted field, retain document ID/version, page, source line or bounding box, raw text and normalized value. The application should assign the document scope. The model may select a source location within it, but must not supply a new document as its own evidence. Amazon Textract's expense response objects are one concrete example of separating field type, detected label and detected value with location/confidence information. A confidence score is an input to evaluation, not a calibrated probability that your accounting entry is correct.

Make normalization an explicit contract

The following synthetic profile supports EUR invoices with non-negative amounts, two decimal places, comma decimals and optional period thousands separators. It validates line-item totals, subtotal, stated tax amount and total. It does not calculate a legally applicable tax rate or validate a Colombian electronic invoice. Credit notes, discounts, withholding, mixed currencies and other layouts require another reviewed profile or human review.

Do not infer a format from a single ambiguous string. 1,234 may mean a decimal value or a grouped integer under different conventions. The source format belongs to the document profile; the normalized extraction format is a separate, fixed representation such as 1234.00.

Python's Decimal documentation explains why decimal strings preserve monetary values more predictably than binary floats. Construct Decimal from strings and define the rounding rule when a workflow actually calculates amounts. This example compares already printed two-decimal amounts exactly; it adds no tolerance that could conceal an extraction error.

Run a source-bound amount gate

Use Python 3.11. There are no dependencies, API keys or model calls. Save these blocks, in order, as validate_invoice.py:

import re
from dataclasses import dataclass, replace
from decimal import Decimal


class ReviewRequired(ValueError):
    pass


@dataclass(frozen=True)
class Amount:
    value: str       # Canonical extraction, for example "1234.56".
    line_id: str     # ID in the application's source document.
    quote: str       # Raw amount, for example "1.234,56".


@dataclass(frozen=True)
class Invoice:
    currency: str
    items: tuple[Amount, ...]
    subtotal: Amount
    tax: Amount
    total: Amount


def checked_amount(amount: Amount, label: str,
                   source: dict[str, str]) -> Decimal:
    if (not isinstance(amount.value, str)
            or not re.fullmatch(r"(?:0|[1-9][0-9]{0,11})\.[0-9]{2}", amount.value)):
        raise ReviewRequired(f"{label}: invalid normalized amount")
    if (not isinstance(amount.quote, str) or len(amount.quote) > 40
            or not re.fullmatch(
                r"(?:0|[1-9][0-9]*|[1-9][0-9]{0,2}(?:\.[0-9]{3})+),[0-9]{2}",
                amount.quote)):
        raise ReviewRequired(f"{label}: unsupported source number format")
    expected_line = f"{label}: EUR {amount.quote}"
    if source.get(amount.line_id) != expected_line:
        raise ReviewRequired(f"{label}: missing or mismatched source line")
    observed = Decimal(amount.quote.replace(".", "").replace(",", "."))
    predicted = Decimal(amount.value)
    if predicted != observed:
        raise ReviewRequired(f"{label}: prediction differs from source")
    return predicted


def validate(invoice: Invoice, source: dict[str, str]) -> None:
    if invoice.currency != "EUR":
        raise ReviewRequired("Unsupported or missing explicit currency")
    if not 1 <= len(invoice.items) <= 100:
        raise ReviewRequired("Unsupported item count")
    fields = (*invoice.items, invoice.subtotal, invoice.tax, invoice.total)
    if len({amount.line_id for amount in fields}) != len(fields):
        raise ReviewRequired("Source line reused for more than one field")
    items = [checked_amount(item, "Item", source) for item in invoice.items]
    subtotal = checked_amount(invoice.subtotal, "Subtotal", source)
    tax = checked_amount(invoice.tax, "Tax", source)
    total = checked_amount(invoice.total, "Total", source)
    if sum(items, Decimal("0.00")) != subtotal:
        raise ReviewRequired("Item totals do not match subtotal")
    if subtotal + tax != total:
        raise ReviewRequired("Subtotal plus tax does not match total")

The source-line check deliberately matches the label, currency and raw value together. Finding 1190.00 anywhere in the document would allow a payment reference or another subtotal to support the wrong field. Reusing a source line for two fields also triggers review, rather than counting one item twice.

The expected line format is intentionally restrictive. Real OCR adapters must preserve trustworthy source locations and define how they map to each profile; do not relax this check by stripping every non-digit character. That would destroy separators and field context. The gate does not prove that every line item was extracted; label and test completeness separately. A complete pipeline also needs validated identifiers, dates and vendor/account matching; passing this amount gate alone does not approve a payment or authorize a ledger write.

Distinguish a wrong prediction from a contradictory invoice

Append a fixture with two line items:

SOURCE = {
    "p1-l1": "Item: EUR 800,00",
    "p1-l2": "Item: EUR 200,00",
    "p1-l3": "Subtotal: EUR 1.000,00",
    "p1-l4": "Tax: EUR 190,00",
    "p1-l5": "Total: EUR 1.190,00",
}

GOOD = Invoice(
    currency="EUR",
    items=(Amount("800.00", "p1-l1", "800,00"),
           Amount("200.00", "p1-l2", "200,00")),
    subtotal=Amount("1000.00", "p1-l3", "1.000,00"),
    tax=Amount("190.00", "p1-l4", "190,00"),
    total=Amount("1190.00", "p1-l5", "1.190,00"),
)


if __name__ == "__main__":
    validate(GOOD, SOURCE)
    print("Amount gate passed")
    wrong_prediction = replace(GOOD, total=replace(GOOD.total, value="1090.00"))
    try:
        validate(wrong_prediction, SOURCE)
    except ReviewRequired as error:
        print(f"Review: {error}")

Run python validate_invoice.py. It prints Amount gate passed and Review: Total: prediction differs from source.

Now change both the source total and its extraction to 1.090,00 / 1090.00. Source agreement passes, but arithmetic fails: subtotal plus tax is still 1190.00. The correct response is review of a contradictory document or an unsupported profile, not silently rewriting the printed total. If OCR changed the original image incorrectly, both checks may agree on the wrong text; inspect the original page in the review interface.

Measure accuracy on reviewed documents

Create a labeled set with the original document, reviewed field values and the source locations that justify them. Separate development documents from the fixed evaluation set. Split by vendor/template and time where possible: near-identical copies on both sides can overstate generalization.

Report at least these measurements with their denominators:

• Field exact match: correct normalized values divided by labeled opportunities for each field. Define amount, date and identifier normalization before scoring.
• Missing-field rate: expected fields that were absent from the extraction. A missing tax is different from a real zero tax.
• Unsupported-value rate: predictions without verified document evidence, including invented fields on documents where the field is absent.
• Document exact match: documents where every required field is correct. High average field accuracy can coexist with many unusable documents.
• Gate coverage: documents passing the proposed automation gate divided by all evaluated documents.
• Error rate among passed documents: passed documents with a material labeled error divided by passed documents. Report it as undefined when none pass.

For a toy illustration, if 20 reviewed documents contain 10 gate passes and two of those passes have a wrong required field, coverage is 10/20 = 50% and error among passes is 2/10 = 20%. These are arithmetic examples, not a model result or a target. A gate that sends everything for review has zero coverage; it has no measured error rate among passed documents.

Review a sample of passes as well as all failures. Otherwise you learn only about rejected documents and never estimate escaped errors. Break results down by scan quality, layout, currency/format, missing fields and multipage documents. Record dataset size and the actual count of errors; a percentage from a tiny sample has substantial uncertainty.

Turn failures into explicit operational states

Keep schema_error, source_mismatch, arithmetic_mismatch, unsupported_profile, provider_error and ready_for_review separate. Repeating a model call may repair an output-shape error; it does not resolve an ambiguous separator or inconsistent invoice. Bound retries and keep the original document and attempted extraction available for review.

The example was exercised with valid amounts, incorrect predictions, missing evidence, repeated lines, unsupported currencies/number formats and arithmetic mismatches. Those checks validate this synthetic profile. They do not measure OCR/model accuracy, tax compliance or a production payment process. You need labeled documents for the former and the relevant accounting rules for the latter.

For a larger pipeline, Prefect orchestration can organize extraction and review tasks. Keep raw invoice content out of routine logs, protect source documents and retain corrections with provenance. If your team needs to move from a document demo to a validated workflow, the AI implementation process covers scoping, implementation and verification.