AI agents running long-duration security tasks produce logs at every step. After a few weeks, those logs exceed the context window. The agent has to read its entire history just to resume work, leaving no room for new reasoning. It restarts, loses intermediate results, and repeats failed attempts.
An AppSec engineer running agents for recon, attack surface mapping, and hypothesis testing hit this wall after a year. The solution was not better compression. It was switching from append-only markdown files to a queryable SQLite database.
The Failure Mode
Markdown logs accumulate fast. A single security project generates files like:
recon.mdsubdomains-live.mdendpoints.mdattempts-auth.mdnotes-session.mdTODO.md,TODO-v2.mdprogress-subagent-3.md
Even with a 256k token context window, the agent eventually cannot fit the full archive. Specific symptoms:
Context compaction erases progress. After compression, the agent re-reads progress.md, sees “invoices module - almost done,” and starts over. What “almost” meant is lost.
Failed attempts repeat. Some hypotheses get tried three times, each time as if it were the first.
Subagents ignore lead agent decisions. Scope constraints live in the main agent’s context. The subagent has no idea.
“What is left?” has no quick answer. You have to read every file and reconstruct the picture yourself.
A markdown file is fine for human reading. As a state store for a running agent, it has no structure, no consistent statuses, and no query interface.
What Structured Storage Fixes
The requirements were clear:
- Restore full context with minimum tokens when starting a session or spawning a subagent.
- Answer “What is left?” with one query.
- Track what has been tried and what failed, without re-parsing thousands of lines.
- Let subagents query shared state without duplicating the entire log.
SQLite solves this. It is a single file, requires no server, and supports concurrent reads. The schema looks like this:
CREATE TABLE tasks (
id INTEGER PRIMARY KEY,
parent_id INTEGER,
description TEXT NOT NULL,
status TEXT CHECK(status IN ('pending', 'in_progress', 'done', 'failed')),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
agent_id TEXT,
result TEXT,
error TEXT
);
CREATE TABLE hypotheses (
id INTEGER PRIMARY KEY,
task_id INTEGER,
hypothesis TEXT NOT NULL,
status TEXT CHECK(status IN ('untested', 'testing', 'confirmed', 'rejected')),
attempts INTEGER DEFAULT 0,
last_attempt TIMESTAMP,
evidence TEXT,
FOREIGN KEY(task_id) REFERENCES tasks(id)
);
CREATE TABLE findings (
id INTEGER PRIMARY KEY,
task_id INTEGER,
finding_type TEXT,
severity TEXT,
description TEXT,
discovered_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
FOREIGN KEY(task_id) REFERENCES tasks(id)
);This schema separates task hierarchy, hypothesis tracking, and security findings. Each table has a clear purpose. The agent can query by status, parent task, or attempt count without loading the full history.
How Agents Query Their Own State
Instead of reading a 50k-token markdown file, the agent runs:
SELECT id, description, status
FROM tasks
WHERE status IN ('pending', 'in_progress')
ORDER BY created_at;This returns only incomplete work. Token cost: a few hundred at most.
To avoid repeating failed hypotheses:
SELECT hypothesis, attempts, error
FROM hypotheses
WHERE status = 'rejected' AND attempts >= 2;This tells the agent what has already been tried and why it failed.
For subagent handoff, the lead agent writes:
INSERT INTO tasks (parent_id, description, status, agent_id)
VALUES (42, 'Enumerate endpoints on subdomain X', 'pending', 'subagent-recon-3');The subagent queries:
SELECT * FROM tasks WHERE agent_id = 'subagent-recon-3' AND status = 'pending';No context duplication. No re-parsing. The subagent gets exactly the work assigned to it.
State Recovery and Session Boundaries
When an agent crashes or hits a timeout, the next session starts with:
SELECT * FROM tasks WHERE status = 'in_progress';These are tasks that were interrupted. The agent can resume or mark them as failed.
For long-running security scans, the database becomes the source of truth. The agent does not need to remember everything. It queries what it needs when it needs it.
Trade-offs and Failure Modes
| Aspect | Markdown Logs | SQLite |
|---|---|---|
| Token cost per resume | Entire log (10k-100k tokens) | Query result only (100-1k tokens) |
| Query by status | Full text search or manual scan | Single SQL statement |
| Concurrent access | File locking issues | Read-heavy workloads work fine |
| Human readability | Excellent | Requires SQL or export script |
| Schema evolution | Freeform, no migration needed | Requires ALTER TABLE or new columns |
| Backup complexity | Copy files | Copy single .db file |
SQLite is not perfect. If the schema changes mid-project, you need a migration script. If the database file corrupts, you lose everything (backups are critical). If you want to read the log as prose, you need to export it.
But for agents that need to query their own execution history, SQLite is the right tool. It turns “read the entire log” into “query the relevant rows.”
Observability and Debugging
With structured storage, you can answer:
- How many hypotheses were tested?
SELECT COUNT(*) FROM hypotheses WHERE status = 'confirmed'; - Which tasks took longest?
SELECT description, updated_at - created_at AS duration FROM tasks ORDER BY duration DESC; - What did subagent-recon-3 actually do?
SELECT * FROM tasks WHERE agent_id = 'subagent-recon-3';
You can export the database to JSON or CSV for external analysis. You can write a simple web UI that shows task progress in real time. You can run analytics on agent behavior without parsing markdown.
Technical Verdict
Use SQLite for agent state when:
- Agents run for days or weeks and accumulate logs that exceed context windows.
- You need to query execution history by status, timestamp, or agent ID.
- Subagents need to share state without duplicating the full context.
- You want to resume work after crashes or timeouts without re-reading everything.
Stick with markdown logs when:
- The agent runs for minutes, not hours.
- You need human-readable output more than queryable structure.
- The log is small enough to fit in context every time.
- You are prototyping and schema changes happen every day.
For long-running security agents, the switch from markdown to SQLite is not optional. It is the difference between an agent that chokes on its own output and one that can actually finish the job.