Enterprise agents hit a token budget wall when reasoning over unstructured data. A single question about scattered evidence in contracts, earnings calls, or web pages can consume a million tokens because the agent reopens large documents repeatedly. If the same data were already structured in a database, the query would be a cheap lookup.
A new ArXiv paper (2608.31082v1) proposes agentic data cracking: structure unstructured data adaptively as a byproduct of reasoning itself. The agent extracts grounded structure speculatively whenever it opens a document, so future related queries hit structured data instead of raw text. On FanOutQA (a multi-document benchmark), this approach cuts costs by 53% while preserving accuracy.
The Token Economics Problem
Agents reason over unstructured sources by loading documents into context, extracting evidence, and synthesizing answers. Each reasoning step reloads the same documents because the agent has no memory of what it extracted before.
Cost breakdown for a typical multi-document query:
- Load 10 PDFs (200 pages each) into context: ~500k tokens
- Extract evidence across 3 reasoning hops: 1.5M tokens total
- Synthesize answer: 50k tokens
If the data were pre-structured (tables, JSON, key-value pairs), the same query becomes:
- Query structured store: 5k tokens
- Synthesize answer: 50k tokens
The FanOutQA benchmark shows a 28X cost reduction when reasoning over ideal pre-structured data. The gap grows to orders of magnitude as queries fan out over more documents.
Why Not Structure Everything Upfront?
Pre-structuring all documents in advance fails for two reasons:
- Structural explosion: A 200-page contract contains vastly more possible structure (tables, entities, relationships, timelines) than any workload will use. Extracting everything is prohibitively expensive.
- Unknown workload: You don’t know which documents or which structure matters until queries arrive. Speculative structuring without query signals wastes compute.
Traditional ETL pipelines solve this by defining schemas upfront, but enterprise unstructured data (web scrapes, filings, call transcripts) resists fixed schemas. The useful structure emerges from the questions people ask.
Agentic Data Cracking Architecture
Agentic data cracking structures data adaptively and speculatively during reasoning. When the agent opens a document to answer a question, a cracking sub-agent forks from the already-loaded context and extracts structure likely to serve related future queries.
Orchestration Flow
# Simplified cracking orchestration
class CrackingAgent:
def __init__(self, llm, structured_store):
self.llm = llm
self.store = structured_store
self.cracking_budget = 0.2 # 20% of query budget
def answer_query(self, query, documents):
# Check if structured data covers query
if self.store.can_answer(query):
return self.store.query(query)
# Load documents into context (expensive)
context = self.load_documents(documents)
# Main reasoning agent extracts answer
answer = self.llm.reason(query, context)
# Fork cracking sub-agent at marginal cost
# (context already loaded, reuse embeddings)
if self.should_crack(query, documents):
self.crack_speculatively(query, context)
return answer
def crack_speculatively(self, query, context):
# Extract structure beyond current query
# - Related entities, dates, amounts
# - Tables, key-value pairs
# - Relationships between documents
structure = self.llm.extract_structure(
query,
context,
budget=self.cracking_budget
)
self.store.insert(structure)Key Decisions
When to crack: The agent cracks when it detects a document will likely be queried again. Signals include:
- Document opened multiple times in recent queries
- Query pattern suggests follow-up questions (e.g., “What were Q3 earnings?” followed by “Compare to Q2”)
- Document type (contracts, filings) with high reuse probability
What to extract: The cracking sub-agent extracts structure grounded in the current query but generalizes beyond it. For a query about revenue, it extracts:
- Revenue tables (current query)
- Cost tables (likely follow-up)
- Date ranges, entities, product lines (context for future queries)
How much to crack: The paper allocates 20% of the query token budget to cracking. This is a marginal cost because the document is already loaded. The sub-agent reuses embeddings and context from the main reasoning agent.
State Management and Versioning
Structured data extracted by cracking must be versioned and invalidated when source documents change.
| Challenge | Solution |
|---|---|
| Source document updated | Hash document content; invalidate cached structure on mismatch |
| Partial structure coverage | Track which queries are fully/partially covered; fall back to raw text for gaps |
| Conflicting extractions | Version structure by extraction timestamp; prefer newer extractions |
| Schema drift | Store structure as flexible JSON; no rigid schema enforcement |
| Query coverage tracking | Log which queries hit structured store vs. raw documents; prioritize cracking for high-miss documents |
The structured store is not a traditional database. It’s a query-adaptive cache that grows as the agent encounters new questions. Each extraction is tagged with:
- Source document hash
- Query that triggered extraction
- Extraction timestamp
- Confidence score
Token Budget Trade-Offs
Cracking introduces upfront structuring overhead in exchange for per-query savings across future reasoning steps.
Break-even analysis (from FanOutQA results):
- First query: 100% cost (no structured data exists yet)
- Second related query: 47% cost (some structure cached)
- Third+ related queries: 20-30% cost (most structure cached)
The paper extends FanOutQA by adding one related question per test question. This simulates real workloads where users ask follow-up questions about the same documents. Cracking cuts total cost by 53% while preserving accuracy.
When cracking fails to pay off:
- One-off queries with no follow-ups
- Documents queried only once
- Queries requiring full-text search (structure doesn’t help)
- Rapidly changing documents (cache invalidation overhead)
Observability and Failure Modes
Cracking agents need instrumentation to detect when structuring is helping vs. wasting tokens.
Key metrics:
- Cache hit rate: % of queries fully answered by structured store
- Partial hit rate: % of queries partially answered (hybrid structured + raw text)
- Cracking ROI: tokens saved on future queries / tokens spent on cracking
- Extraction accuracy: % of cracked structure that is actually used in future queries
Failure modes:
- Over-cracking: Extracting structure that is never queried. Mitigate by tracking extraction usage and adjusting cracking heuristics.
- Under-cracking: Missing structure that would serve future queries. Mitigate by analyzing cache misses and expanding extraction scope.
- Stale structure: Cached structure becomes outdated when source documents change. Mitigate by hashing documents and invalidating on mismatch.
- Incorrect extraction: Cracking sub-agent hallucinates structure. Mitigate by grounding extractions in source text spans and logging confidence scores.
Deployment Shape
Agentic data cracking requires infrastructure beyond a stateless LLM API.
Required components:
- Structured store: Vector DB or hybrid search index (e.g., Weaviate, Qdrant) that stores extracted structure alongside source document references
- Cracking orchestrator: Decides when to crack, forks sub-agents, manages token budgets
- Cache invalidation service: Monitors source documents for changes, invalidates stale structure
- Observability layer: Tracks cache hit rates, cracking ROI, extraction accuracy
Deployment options:
- Embedded: Cracking logic runs in the same process as the main agent (low latency, tight coupling)
- Sidecar: Cracking sub-agent runs as a separate service (better isolation, harder to reuse context)
- Async: Cracking happens asynchronously after the main query returns (lower latency for first query, delayed benefit)
Security Boundaries
Cracking introduces new attack surfaces because extracted structure is cached and reused across queries.
Threats:
- Poisoned extractions: Attacker injects malicious structure into the cache by crafting adversarial queries
- Cache inference attacks: Attacker infers sensitive data by observing cache hit patterns
- Stale data leakage: Cached structure persists after source document is deleted or access-controlled
Mitigations:
- Scope structured stores per user or tenant (no cross-tenant cache sharing)
- Tag extractions with access control metadata from source documents
- Invalidate cache entries when source document permissions change
- Log all cache hits for audit trails
Technical Verdict
Use agentic data cracking when:
- Users ask follow-up questions about the same documents (contracts, filings, reports)
- Token costs are a primary constraint (high query volume, large documents)
- Documents are relatively stable (updated weekly or monthly, not real-time)
- You can afford infrastructure for structured storage and cache invalidation
Avoid when:
- Queries are one-off with no follow-ups (no amortization of cracking cost)
- Documents change frequently (cache invalidation overhead dominates)
- Queries require full-text search or nuanced interpretation (structure doesn’t help)
- You need deterministic, auditable reasoning (cracking introduces non-determinism)
The 53% cost reduction on FanOutQA is compelling, but the benchmark simulates ideal conditions (related queries, stable documents). Real-world gains depend on workload characteristics. Instrument cache hit rates and cracking ROI before committing to this architecture.