Midship's Document Extraction API: How Agents Turn Unstructured PDFs into Structured Tool Inputs

Document extraction APIs sit at a critical boundary in agent workflows. An agent that processes invoices, analyzes contracts, or automates form submissions needs structured JSON from unstructured PDFs. When the extraction fails or returns malformed data, the entire pipeline stalls. Midship (YC S24) launched with 121 HN points and a live demo showing PDF-to-JSON extraction, but the interesting plumbing question is not whether it works. It is how agents handle schema drift, validation failures, and confidence thresholds when documents do not match expectations.

The Tool Boundary Problem

Agents call tools. Tools expect typed inputs. PDFs are chaos: scanned images with OCR noise, layout variations between vendors, handwritten annotations, multi-column tables that break parsing logic. The extraction API becomes a translation layer, and someone has to own the contract.

Three handoff points where things break:

Schema negotiation: Agent defines expected fields (invoice_number, line_items, total). Extraction returns what it finds. Mismatch causes downstream tool calls to fail.
Validation layer: Who checks that extracted data meets constraints? If the API returns a date string but the agent’s database tool expects ISO 8601, the error surfaces three steps later.
Confidence thresholds: OCR on a blurry scan might return “Inv# 12O45” (letter O instead of zero). Does the agent retry, escalate to human review, or proceed with dirty data?

Most extraction APIs return JSON and call it done. The agent is left holding the bag when field names drift, values are malformed, or confidence scores are missing.

How Midship Positions the Plumbing

Midship combines OCR with LLMs to extract specific fields and tables. The LLM layer corrects OCR mistakes and normalizes field names (understanding that “Inv#” and “Invoice Number” are equivalent). This is useful, but the architecture question is where validation happens.

Likely flow:

Agent uploads PDF via API
OCR extracts raw text and layout
LLM maps text to user-defined schema
API returns JSON with extracted fields
Agent validates and passes to next tool

The gap: what happens when step 3 produces low-confidence results or step 5 fails validation? Does the API expose confidence scores per field? Can the agent request a retry with a refined prompt? Is there a webhook for async processing when documents are large?

Schema Drift and Version Control

Document layouts change. A vendor updates their invoice template. A contract form adds a new section. The agent’s schema is now stale.

Strategies for handling drift:

Approach	Trade-off	When to Use
Strict schema validation	Fails fast, requires manual schema updates	High-value workflows where bad data is worse than downtime
Flexible extraction with confidence scores	Proceeds with partial data, logs low-confidence fields	Batch processing where human review happens later
Schema versioning with fallback	Tries new schema, falls back to old on failure	Gradual migration across document sources
LLM-driven schema inference	Adapts to new layouts automatically, unpredictable	Exploratory workflows, low compliance requirements

Midship’s LLM layer likely helps with minor variations (different field labels, reordered sections), but agents still need a way to detect when extraction quality degrades. Observability hooks matter: per-field confidence, extraction latency, retry counts.

Error Recovery in Multi-Step Pipelines

An agent processing 500 invoices hits a malformed PDF at document 237. What happens?

Recovery patterns:

Fail the batch: Stop processing, alert operator. Simple but wasteful.
Skip and log: Continue with remaining documents, flag failures for review. Requires idempotent downstream tools.
Retry with fallback: Attempt extraction again with different OCR settings or prompt. Adds latency and cost.
Human-in-the-loop escalation: Route low-confidence extractions to review queue. Breaks full automation but prevents bad data propagation.

The API needs to surface enough metadata for the agent to make this decision. A boolean success flag is not enough. The agent needs per-field confidence, OCR quality indicators, and extraction duration to implement smart retry logic.

Rate Limiting and Cost Control

Agents loop. An invoice processing agent might extract 10,000 documents per month. If each extraction costs $0.10 and the agent retries on failures, costs spiral.

Cost control mechanisms:

Client-side rate limiting: Agent enforces max requests per minute. Prevents runaway loops but requires state management.
API-side quotas: Hard limits per API key. Protects the service but causes agent failures when hit.
Tiered pricing with burst allowances: Pay-as-you-go with monthly caps. Balances flexibility and predictability.
Batch endpoints: Submit 100 documents, get async results. Reduces per-request overhead.

Midship charges volume-based pricing for the API. The question is whether the pricing model includes retry budgets or penalizes agents that implement quality checks.

Observability Hooks for Debugging

When extraction fails in a multi-step workflow, the agent needs to know why. Was it OCR quality? Schema mismatch? LLM hallucination?

Useful telemetry:

Per-field extraction confidence (0-1 score)
OCR quality metrics (contrast, resolution, text density)
LLM prompt and response for each extraction
Processing duration breakdown (OCR time, LLM time, validation time)
Retry attempts and outcomes

Without this, debugging becomes guesswork. The agent logs “extraction failed” and the operator manually inspects the PDF, re-runs the extraction, and hopes for different results.

Code Example: Agent Validation Layer

Here is how an agent might validate extraction results before passing to downstream tools:

import requests
from pydantic import BaseModel, ValidationError
from typing import Optional

class InvoiceSchema(BaseModel):
    invoice_number: str
    date: str  # ISO 8601
    total: float
    line_items: list[dict]
    confidence: Optional[float] = None

def extract_and_validate(pdf_path: str, api_key: str):
    # Call extraction API
    with open(pdf_path, 'rb') as f:
        response = requests.post(
            'https://api.midship.ai/extract',
            headers={'Authorization': f'Bearer {api_key}'},
            files={'document': f},
            json={'schema': InvoiceSchema.schema()}
        )
    
    result = response.json()
    
    # Validate against expected schema
    try:
        invoice = InvoiceSchema(**result['data'])
    except ValidationError as e:
        # Log validation failure, decide on retry or escalation
        if result.get('confidence', 1.0) < 0.7:
            return retry_with_manual_review(pdf_path)
        else:
            raise e
    
    # Check confidence threshold
    if invoice.confidence and invoice.confidence < 0.85:
        log_low_confidence(invoice)
        # Proceed but flag for audit
    
    return invoice

This pattern puts validation in the agent, not the API. The agent owns the contract and decides how to handle failures.

Technical Verdict

Use Midship-style extraction APIs when:

You have high-volume document workflows where manual extraction is a bottleneck
Document layouts are semi-structured (invoices, forms, contracts) with predictable fields
You can implement validation and retry logic in your agent orchestration layer
You need field-level extraction, not just full-text OCR

Avoid or supplement when:

Documents are highly variable (handwritten notes, artistic layouts, multi-language)
You need guaranteed accuracy for compliance (financial reporting, legal discovery)
Your agent cannot handle partial failures gracefully
Cost per extraction exceeds the value of automation (low-volume, high-touch workflows)

The real test is not whether the API extracts data. It is whether your agent can recover when extraction is ambiguous, validate results before acting, and surface failures without breaking the pipeline. Document extraction is a tool boundary problem, and the API is only half the solution. The other half is orchestration logic that handles the messy reality of PDFs in production.

Agentic AI, minus the noise.