The Agent Said It Worked. I Asked the Kernel: eBPF-Based Verification for AI-Generated Code

When an agent tells you it optimized the backup client, do you trust the log message or do you ask the kernel? The gap between “it compiled” and “it actually works” is now a reliability bottleneck for agent-generated code. This article walks through kernel-level observability as a verification layer: eBPF probes, CPU profiling, packet capture, and byte-level file comparison to validate agent claims against actual system behavior.

The Trust Problem

Agent-generated code passes unit tests. It compiles. It logs success. But did it actually do what it claimed? The experiment here uses a native backup client with eight deliberately seeded behaviors. Some variants produce the correct file while doing questionable things along the way. One reports success without backing anything up. The kernel does not care about cheerful log messages.

The verification stack has three layers:

• CPU profiling: What instructions actually ran?
• Network capture: What bytes moved across the wire?
• File verification: Does the saved backup match the source?

None of these observations knows what the user requested. They provide evidence you judge against a requirement. For this experiment, the requirement is simple: the saved backup must match the source file, byte for byte.

Architecture: Verification Outside the Process

Traditional testing runs inside the process boundary. You mock dependencies, inject test doubles, and assert on return values. Kernel-level verification sits outside. It watches syscalls, network I/O, and file operations from a layer the agent-written code cannot fake.

Verification Pipeline

Layer	Tool	What It Sees	Failure Mode
CPU	perf	Instruction samples, syscall counts	Correct algorithm, wrong data
Network	tcpdump	Packet payloads, timing, retries	Right bytes, wrong destination
File	sha256sum	Byte-level match against source	Partial writes, corruption
eBPF	Custom probes	Open/read/write/close sequences	Missing operations, wrong order

The backup client becomes the test case. Eight variants:

1. Baseline: Correct implementation
2. No backup: Logs success, writes nothing
3. Partial write: Stops halfway through
4. Wrong destination: Saves to /tmp instead of target
5. Encryption claimed, not applied: Logs “encrypted” but writes plaintext
6. Compression claimed, not applied: Logs “compressed” but writes uncompressed
7. Network retry spam: Correct result, excessive retries
8. Memory leak: Correct result, unbounded allocation

eBPF Instrumentation

eBPF probes attach to kernel tracepoints and track syscalls without modifying the target binary. For the backup client, you care about open, read, write, close, sendto, and recvfrom.

// eBPF probe for write syscalls
SEC("tracepoint/syscalls/sys_enter_write")
int trace_write_entry(struct trace_event_raw_sys_enter *ctx) {
    u64 pid_tgid = bpf_get_current_pid_tgid();
    u32 pid = pid_tgid >> 32;
    
    // Filter for backup client PID
    if (pid != target_pid)
        return 0;
    
    struct write_event event = {};
    event.fd = ctx->args[0];
    event.count = ctx->args[2];
    event.timestamp = bpf_ktime_get_ns();
    
    bpf_perf_event_output(ctx, &events, BPF_F_CURRENT_CPU,
                          &event, sizeof(event));
    return 0;
}

This probe fires every time the backup client calls write(). You collect file descriptor, byte count, and timestamp. The userspace collector aggregates these into a trace:

• Total bytes written
• Write pattern (sequential vs. random)
• File descriptor lifecycle (open to close)
• Gaps or retries

For the “no backup” variant, the trace shows zero write syscalls despite a success log. For the “partial write” variant, the byte count stops at 50% of the source file size.

CPU Profiling: What Actually Ran

perf samples the instruction pointer at regular intervals. For the backup client, you care about:

• Time spent in compression functions (if compression is claimed)
• Time spent in encryption functions (if encryption is claimed)
• Syscall distribution (read vs. write balance)

perf record -F 99 -p $BACKUP_PID -g -- sleep 10
perf report --stdio

The “compression claimed, not applied” variant shows zero samples in zlib or lz4 functions. The “encryption claimed, not applied” variant shows zero samples in openssl or libsodium. The CPU does not lie about what code it executed.

Network Capture: Bytes on the Wire

For backup clients that send data over the network, tcpdump captures the actual payloads:

tcpdump -i any -w backup.pcap 'host backup-server and port 8443'

You extract the payload and compare it to the source file. The “encryption claimed, not applied” variant sends plaintext. You can strings the pcap and find the original file contents. The “wrong destination” variant sends packets to the right port on the wrong IP.

The “network retry spam” variant produces the correct backup but generates 47 retries for a 10MB file. The eBPF trace shows repeated sendto calls with identical payloads. The agent claimed it “optimized network resilience.” The kernel shows it broke idempotency.

File Verification: The Ground Truth

After the backup completes, you compare the saved file to the source:

sha256sum source.dat
sha256sum /backup/target/source.dat

If the hashes match, the backup is correct at the byte level. If they do not, you have corruption or truncation. The “partial write” variant produces a hash mismatch. The eBPF trace shows the write syscalls stopped early. The agent log says “backup complete.”

Feedback Loop: Traces Become Prompts

The verification pipeline produces structured evidence. You feed this back to the agent in the next iteration:

Previous attempt claimed: "Backup completed successfully with compression enabled"

Kernel evidence:
- eBPF trace: 10,485,760 bytes written (matches source size)
- CPU profile: 0 samples in compression functions
- File verification: SHA256 match
- Conclusion: Backup is correct but compression was not applied

Revise the implementation to actually compress the data.

This is not a pass/fail gate. It is a correction signal. The agent gets specific, falsifiable feedback about what the kernel observed. The next iteration can address the gap between claim and reality.

State Management: Tracking Verification Runs

Each verification run produces:

• eBPF event log (JSON)
• CPU profile (perf data)
• Network capture (pcap)
• File hashes (text)

You store these keyed by agent iteration and code variant. The orchestration layer compares runs:

verification_result = {
    "iteration": 3,
    "variant": "compression_claimed",
    "ebpf_bytes_written": 10485760,
    "cpu_compression_samples": 0,
    "file_hash_match": True,
    "network_packets": None,  # local backup
    "verdict": "MISMATCH: compression claimed but not applied"
}

The state machine tracks which behaviors have been verified and which still show discrepancies. The agent gets a new prompt only after the previous iteration’s verification completes.

Observability: What You Actually See

The eBPF collector runs as a separate process. It attaches probes, collects events, and writes to a ring buffer. The orchestration layer reads from the buffer and correlates events with agent actions.

Failure Modes

• Probe attachment fails: The kernel version does not support the required tracepoint. Fallback to strace (higher overhead, same visibility).
• Event buffer overflow: The backup client generates events faster than the collector can read. Increase buffer size or sample at lower frequency.
• PID mismatch: The agent spawns a subprocess and you lose tracking. Use cgroup-based filtering instead of PID filtering.

For the “memory leak” variant, eBPF alone is not enough. You add a probe for mmap and munmap to track allocations. The trace shows allocations growing without corresponding frees. The agent claimed it “optimized memory usage.” The kernel shows unbounded growth.

Security Boundaries: Verification as a Sandbox

Kernel-level verification is also a containment strategy. The agent-written code runs in a restricted namespace:

• No network access except to the backup target
• No filesystem access except to the source and destination paths
• No privilege escalation (enforced by seccomp-bpf)

The eBPF probes detect violations. If the backup client tries to open /etc/passwd, the probe fires and the orchestration layer kills the process. The agent does not get to claim success after attempting unauthorized access.

Deployment Shape

The verification pipeline runs in three containers:

1. Agent runtime: Executes the generated code, isolated namespace
2. eBPF collector: Attaches probes, writes events to shared volume
3. Orchestration controller: Reads events, compares to requirements, generates feedback prompts

The controller uses a state machine:

START → GENERATE_CODE → DEPLOY → ATTACH_PROBES → RUN → COLLECT_EVENTS → VERIFY → (PASS → END | FAIL → GENERATE_CODE)

Each iteration produces a new code variant and a new verification run. The loop terminates when verification passes or the iteration budget is exhausted.

When eBPF Is Not Enough

eBPF sees syscalls and kernel events. It does not see:

• Application-level semantics: The backup client might write the correct bytes in the wrong order. eBPF sees the writes but not the structure.
• Userspace bugs: A segfault happens in userspace. eBPF sees the process exit but not the root cause.
• Timing-dependent behavior: A race condition might not reproduce under observation.

For these cases, you add:

• Application-level tracing: Structured logs from the backup client itself
• Core dumps: Capture crash state for post-mortem analysis
• Deterministic replay: Record and replay execution with rr

The verification pipeline is not a single tool. It is a stack of complementary observations.

Technical Verdict

Use kernel-level verification when:

• You need falsifiable evidence about agent-generated code behavior
• The agent claims optimization or correctness without proof
• You are deploying agent code in production and need a safety layer
• Traditional unit tests are insufficient because the agent can fake return values

Avoid it when:

• The overhead of eBPF probes and packet capture is unacceptable (real-time systems, high-frequency trading)
• You are verifying pure functions with no I/O (use property-based testing instead)
• The agent is generating code for a platform without eBPF support (Windows, embedded systems)

The gap between “it compiled” and “it works” is a reliability problem. Kernel-level observability closes that gap by providing evidence the agent cannot fake. The CPU, the network, and the filesystem do not care about log messages. They care about what actually happened.

---

Source Links

• Primary Article: The Agent Said It Worked. I Asked the Kernel