Why Fully Automated Extraction Fails

Every document extraction project starts with the same pitch: upload a PDF, get structured JSON, never look at the document again. It works in the demo. It falls apart in production.

The problem isn't that AI extraction is inaccurate — it's that it's inconsistently accurate. A well-structured invoice from a regular supplier extracts perfectly. A scanned contract with coffee stains and handwritten annotations does not. And when the extraction is wrong, you don't find out until downstream: a wrong invoice total breaks a payment run, a wrong contract date triggers incorrect compliance alerts, a wrong address sends a shipment to the wrong city.

Fully automated pipelines without human oversight don't survive contact with messy real-world documents. But the opposite extreme — manual review of every extracted field — defeats the purpose of automation entirely.

The answer is a human-in-the-loop approach: automate the cases where the AI is reliably correct, and route the uncertain ones to a human reviewer. The question is how to tell the difference. That's where confidence scores come in. Per-field confidence scores give your pipeline a built-in uncertainty signal — a way to separate the extractions the model is sure about from the ones it's guessing at. Instead of choosing between "trust everything" and "review everything," you build a review flow that puts human attention exactly where it matters.

This is the architecture that makes document extraction work at scale: not pure automation, and not pure manual review, but a calibrated loop where humans handle the edge cases and the system handles the rest.

What Confidence Scores Measure

The Iteration Layer's Document Extraction API returns a confidence score between 0.0 and 1.0 for every extracted field. The score reflects how certain the extraction model is about a specific value given the source document.

A confidence of 0.97 means the model found a clear, unambiguous value. A confidence of 0.72 means the model extracted something, but the source was degraded, ambiguous, or formatted in an unexpected way. A confidence of 0.35 means the model is guessing.

Confidence is not accuracy. A field with 0.95 confidence can still be wrong — the model was very sure about an incorrect value. But across thousands of extractions, higher confidence correlates strongly with correctness. A 0.95 field is right far more often than a 0.65 field.

Factors that affect confidence:

• Scan quality — blurry, skewed, or low-resolution scans reduce confidence
• Document structure — well-structured invoices extract at higher confidence than free-form letters
• Field ambiguity — a single clearly labeled "Total" extracts at higher confidence than a table with multiple subtotals
• Handwriting — handwritten values have lower confidence than printed text
• Language mixing — documents mixing languages or scripts reduce confidence for affected fields

Per-Field vs. Document-Level Confidence

Iteration Layer provides confidence per field, not per document. This is a deliberate design decision that changes how you build your pipeline.

A single invoice might return:

{
  "invoiceNumber": {
    "type": "TEXT",
    "value": "INV-2026-4521",
    "confidence": 0.97,
    "citations": [
      "Invoice #INV-2026-4521"
    ],
    "source": "invoice.pdf"
  },
  "vendorName": {
    "type": "TEXT",
    "value": "Meridian Supply Co.",
    "confidence": 0.94,
    "citations": [
      "Meridian Supply Co."
    ],
    "source": "invoice.pdf"
  },
  "totalAmount": {
    "type": "CURRENCY_AMOUNT",
    "value": 3847.50,
    "confidence": 0.96,
    "citations": [
      "Total: $3,847.50"
    ],
    "source": "invoice.pdf"
  },
  "shippingAddress": {
    "type": "ADDRESS",
    "value": {
      "street": "42 Innovation Drive",
      "city": "Austin",
      "region": "TX",
      "postal_code": "78701",
      "country": "US"
    },
    "confidence": 0.72,
    "citations": [
      "42 Innovation Drive, Austin, TX 78701"
    ],
    "source": "invoice.pdf"
  }
}

The invoice number, vendor name, and total are high confidence. The shipping address is lower — maybe the scan was smudged in that area, or the layout was ambiguous. With per-field confidence, you auto-accept the three reliable fields and only route the address for human review. Without per-field confidence, you'd have to review the entire document for one uncertain field.

This is dramatically more efficient. A human reviewer sees one pre-filled field that needs confirmation instead of re-checking every field on the page.

Threshold Strategies

The most direct use of confidence scores is threshold-based routing. Define boundaries and route each field accordingly.

Three-Tier Routing

The most common pattern splits fields into three buckets:

• Auto-accept — confidence is high enough to write directly to your database
• Review — confidence is in a gray zone; pre-fill the value and ask a human to confirm or correct
• Manual entry — confidence is too low to be useful; ask a human to enter the value from scratch

The threshold values depend on your domain and the cost of errors. Here's a starting point:

Financial data (payment amounts, tax calculations) warrants higher thresholds — 0.95 or above for auto-accept. Content aggregation pipelines can tolerate lower thresholds — 0.85 for auto-accept might be fine when the cost of a wrong value is a minor inconvenience, not a financial discrepancy.

Per-Field Thresholds

Not every field deserves the same threshold. A wrong invoice number is annoying but correctable. A wrong payment amount triggers a wrong payment. Set thresholds based on the business cost of getting each field wrong.

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const THRESHOLD_BY_FIELD: Record<string, number> = {
  invoiceNumber: 0.90,
  vendorName: 0.88,
  invoiceDate: 0.90,
  subtotal: 0.95,
  taxAmount: 0.95,
  totalDue: 0.95,
};

const DEFAULT_THRESHOLD = 0.90;

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/INV-2026-4521.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "TEXT",
        name: "invoiceNumber",
        description: "The invoice number or identifier",
      },
      {
        type: "TEXT",
        name: "vendorName",
        description: "The name of the vendor or supplier",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "subtotal",
        description: "The subtotal before tax",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "taxAmount",
        description: "The tax amount",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due including tax",
      },
    ],
  },
});

const routedFields = Object.entries(result).map(
  ([fieldName, fieldResult]) => {
    const threshold =
      THRESHOLD_BY_FIELD[fieldName] ?? DEFAULT_THRESHOLD;

    if (fieldResult.confidence >= threshold) {
      return {
        action: "accept" as const,
        field: fieldName,
        value: fieldResult.value,
        confidence: fieldResult.confidence,
      };
    }

    if (fieldResult.confidence >= 0.70) {
      return {
        action: "review" as const,
        field: fieldName,
        value: fieldResult.value,
        confidence: fieldResult.confidence,
      };
    }

    return {
      action: "manual" as const,
      field: fieldName,
      value: null,
      confidence: fieldResult.confidence,
    };
  },
);

Financial fields (subtotal, tax, total) get a 0.95 bar. Descriptive fields (vendor name) get 0.88. This reflects the real-world cost of errors — a wrong vendor name is a minor annoyance, a wrong total is a financial discrepancy.

Building a Human-in-the-Loop Review Flow

Threshold routing is only useful if the "review" tier actually reaches a human. Here's how to build that.

The Review Queue

A review queue entry should contain everything the reviewer needs to make a decision without leaving the page:

• The original document (as a link, thumbnail, or embedded viewer)
• All extracted fields with their values and confidence scores
• A visual indicator (green, yellow, red) based on your thresholds
• The ability to confirm or correct each flagged field individually
• A single "approve all" action for fields that don't need changes

The reviewer's workflow: scan the flagged fields, confirm or correct each one, approve the document. Fields that were auto-accepted are visible but not editable unless the reviewer explicitly overrides them.

Designing for Speed

The goal of a review queue is to make human review as fast as possible, not to eliminate it. A reviewer who sees one pre-filled field needing confirmation spends seconds, not minutes. Optimize for the common case — most flagged fields are correct and just need a click to confirm.

Patterns that help:

• Pre-fill everything. Even low-confidence fields should show the extracted value. A reviewer who corrects a pre-filled value is faster than a reviewer who types from scratch.
• Show the source. Display the citation next to each field — the exact text the model extracted from. The reviewer can compare the extracted value to the source without reading the full document.
• Keyboard navigation. Tab between fields, Enter to confirm, type to correct. The reviewer should never need a mouse for the common path.
• Batch approval. If all flagged fields look correct, one click approves the entire document.

Routing the Review Result

Once the reviewer confirms or corrects a field, route the result back into your pipeline:

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

// After human review, generate a summary report
const report = await client.generateDocument({
  format: "pdf",
  document: {
    metadata: {
      title: `Invoice Summary - ${reviewedData.invoiceNumber}`,
    },
    content: [
      {
        type: "headline",
        level: "h1",
        text: "Invoice Summary",
      },
      {
        type: "table",
        header: {
          cells: [
            { text: "Field" },
            { text: "Value" },
            { text: "Status" },
          ],
        },
        rows: Object.entries(reviewedData).map(
          ([fieldName, fieldValue]) => ({
            cells: [
              { text: fieldName },
              { text: String(fieldValue) },
              { text: "Verified" },
            ],
          }),
        ),
      },
    ],
  },
});

This is the composability payoff. The extraction feeds into a review queue. The reviewed data feeds into document generation. Same API key, same credit pool, no glue code between services.

Confidence-Based Routing for Agents

If you're building AI agents that process documents — an MCP-connected assistant, a Claude-powered pipeline, a custom agent framework — confidence scores become the agent's decision layer.

An agent without confidence awareness is dangerous. It extracts data and acts on it, with no way to know when the extraction was unreliable. An agent with confidence awareness can make nuanced decisions: proceed when confident, ask for help when uncertain, reject when the data is too unreliable to use.

Agent Decision Patterns

Pattern 1: Proceed or escalate. The agent extracts document data. If all fields are above the threshold, it continues the workflow. If any field falls below, it pauses and escalates to a human.

Pattern 2: Confidence-gated branching. The agent takes different actions based on confidence. High-confidence invoices go straight to payment processing. Medium-confidence invoices get queued for review. Low-confidence invoices get flagged with a note explaining which fields are uncertain.

Pattern 3: Multi-source validation. The agent extracts the same field from multiple documents (e.g., a contract and its amendment). If both extractions agree and both have high confidence, the agent trusts the result. If they disagree, it flags the discrepancy.

MCP Tool Integration

When the Iteration Layer MCP server processes a document, the agent receives the full confidence data alongside the extracted values. The agent can inspect confidence per field and decide its next action without custom parsing logic.

A typical agent workflow:

1. Agent receives a document to process
2. Agent calls the extraction tool via MCP
3. Agent checks confidence scores on each field
4. For high-confidence fields: continues with the workflow
5. For low-confidence fields: asks the user to verify, or flags the document

The confidence scores give the agent the same decision-making framework a human operator would use — but faster, and consistently applied across every document.

Monitoring Confidence Over Time

Confidence scores aren't just a per-document decision tool. They're a signal about the health of your entire pipeline.

What to Track

• Average confidence per field — trending across all documents processed
• Confidence distribution — what percentage of fields fall into each tier (accept, review, manual)
• Auto-accept rate — what percentage of fields pass your threshold without human review
• Review-to-correction rate — how often reviewers actually change the suggested value

Setting Up a Confidence Dashboard

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const HIGH_CONFIDENCE_THRESHOLD = 0.92;
const LOW_CONFIDENCE_THRESHOLD = 0.70;

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/batch-001.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "TEXT",
        name: "invoiceNumber",
        description: "The invoice number",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due",
      },
    ],
  },
});

const confidenceStats = Object.entries(result).map(
  ([fieldName, fieldResult]) => ({
    field: fieldName,
    confidence: fieldResult.confidence,
    tier:
      fieldResult.confidence >= HIGH_CONFIDENCE_THRESHOLD
        ? "accept"
        : fieldResult.confidence >= LOW_CONFIDENCE_THRESHOLD
          ? "review"
          : "manual",
  }),
);

// Send to your monitoring system
console.log(JSON.stringify(confidenceStats, null, 2));

Reading the Signals

Handling Low-Confidence Fields Gracefully

Not every low-confidence field is a failure. Some fields are inherently harder to extract — handwritten notes, freeform text blocks, nested tables with inconsistent formatting. Your system needs to handle these cases without breaking.

Fallback Strategies

Strategy 1: Default values. For optional fields where a wrong value is better than no value, use a sensible default when confidence is below your threshold. The field config supports default_value for this purpose.

Strategy 2: Skip and notify. For non-critical fields, skip the low-confidence extraction entirely and log it. Process the document with the fields you do have. A missing vendor name doesn't block invoice processing if you have the invoice number and amount.

Strategy 3: Re-extract with a different schema. If a field consistently extracts at low confidence, the problem might be the schema description, not the document. Try rewording the field description to be more specific. "The total amount due after tax, displayed at the bottom of the invoice" often extracts better than "total."

Strategy 4: Multi-document cross-reference. For critical fields, extract the same value from multiple related documents. If the invoice total matches the purchase order total and both have high confidence, you have strong validation. If they disagree, flag both for review.

Calculated Fields as Validation

The Document Extraction API supports CALCULATED fields that compute values from other extracted fields. This is a built-in cross-check.

Extract subtotal, taxAmount, and totalDue as separate fields. Add a calculated field that sums subtotal and taxAmount. If the calculated sum doesn't match the extracted totalDue, at least one of the three fields is wrong — flag all three for review, regardless of their individual confidence scores.

This pattern catches a class of errors that confidence alone can't: values that look plausible individually but are internally inconsistent.

Adjusting Thresholds Over Time

Your initial thresholds are educated guesses. After processing a few hundred documents, you have data to make them precise.

The Calibration Loop

1. Start conservative. Set auto-accept thresholds high (0.95) and review thresholds generous (0.65). You'll review more documents than necessary, but you won't miss errors.

2. Track reviewer actions. For every reviewed field, record whether the reviewer confirmed the value or corrected it.

3. Analyze at the 500-document mark. For each field, calculate:

   • What percentage of reviewed fields were confirmed without changes?
   • What was the average confidence of confirmed-without-changes fields?
   • What was the average confidence of corrected fields?

4. Adjust thresholds. If 98% of fields in the review tier were confirmed without changes, your auto-accept threshold is too high. Lower it until the review tier catches roughly 5-10% corrections — that's the sweet spot where review is adding value without wasting time.

5. Repeat quarterly. Document formats change, suppliers change, scan quality changes. Recalibrate regularly.

A Practical Example

After processing 1,000 invoices with a 0.92 auto-accept threshold:

That 96% confirmation rate means your review threshold is too conservative — most flagged fields are correct. Drop the auto-accept threshold to 0.88 and re-measure. The goal: push the auto-accept rate up while keeping the correction rate in the review tier meaningful.

The Business Case for Confidence-Based Automation

Manual document processing costs time. Full automation without verification costs trust. Confidence-based automation gives you both — speed for the clear cases, human oversight for the ambiguous ones.

For a team processing 500 invoices per month with 8 fields per invoice — that's 4,000 field decisions per month. With confidence-based routing:

Compare that to fully manual processing at roughly 2 minutes per field: 133 hours per month. Or fully automated with no review: zero hours, but a 5-15% error rate that shows up as payment discrepancies, compliance issues, and client complaints.

Three hours of targeted review per month. That's what confidence scores buy you.

Cross-Check: Combining Confidence with Calculated Fields

One of the most effective validation patterns combines confidence scores with the CALCULATED field type. This catches errors that look plausible in isolation but fail basic arithmetic.

import { IterationLayer } from "iterationlayer";

const client = new IterationLayer({
  apiKey: process.env.ITERATION_LAYER_API_KEY!,
});

const result = await client.extractDocument({
  files: [
    {
      type: "url",
      name: "invoice.pdf",
      url: "https://example.com/invoices/INV-2026-4521.pdf",
    },
  ],
  schema: {
    fields: [
      {
        type: "CURRENCY_AMOUNT",
        name: "subtotal",
        description: "The subtotal before tax",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "taxAmount",
        description: "The tax amount",
      },
      {
        type: "CURRENCY_AMOUNT",
        name: "totalDue",
        description: "The total amount due",
      },
      {
        type: "CALCULATED",
        name: "computedTotal",
        description: "Sum of subtotal and tax for validation",
        operation: "sum",
        source_field_names: ["subtotal", "taxAmount"],
      },
    ],
  },
});

const extractedTotal = result.totalDue.value as number;
const computedTotal = result.computedTotal.value as number;
const totalsMatch = Math.abs(extractedTotal - computedTotal) < 0.01;

if (!totalsMatch) {
  // Flag all financial fields for review, regardless of confidence
  console.log("Arithmetic mismatch detected — flagging for review");
}

If the extracted total doesn't match the computed sum, flag all three fields for review — even if each individual field has high confidence. The mismatch tells you something is wrong, even when the model is confident about each part.

Practical Domain Patterns

Invoice Processing

Auto-accept: invoice number, date, vendor name (usually high confidence from structured invoices). Flag: line item totals when below threshold. Use CALCULATED fields to cross-check subtotal + tax = total. Financial fields get a 0.95 threshold; descriptive fields get 0.88.

Contract Analysis

Auto-accept: party names, effective dates, contract ID. Flag: clause summaries (TEXTAREA fields have more room for partial extraction). Flag boolean fields like "has non-compete clause" when confidence is below 0.85 — the business impact of a wrong answer is high.

Resume Screening

Auto-accept: name and email (high confidence from structured headers). Flag: skills and experience summaries (often medium confidence due to varied formatting across resumes). Require manual review for contact details extracted from scanned or photographed documents.

Logistics Documents

Auto-accept: shipment ID, origin, destination (usually structured and consistent). Flag: weight, dimensions, declared value (format varies widely across carriers). Use CALCULATED fields to validate that itemized weights sum to the declared total weight.

Get Started

Check the Document Extraction docs to see how confidence scores work across all field types. The TypeScript, Python, and Go SDKs return typed response objects with confidence scores on every field.

Start with conservative thresholds. Process a few hundred documents. Track what your reviewers actually change. Then tighten the thresholds based on data, not guesses.

Sign up for a free account — no credit card required. Run a few documents through with your schema and check the confidence distributions before building your thresholding logic.