Agent loops burn tokens and clock cycles on repeated work. The same LLM prompt fires twice, the same tool call fetches identical data, and session state gets reconstructed from scratch on every request. Agent-cache solves this with a three-tier caching architecture backed by Valkey or Redis, putting LLM responses, tool results, and session snapshots behind a single connection.
The project shipped v0.1.0 with Valkey 7+ and Redis 6.2+ support, then v0.2.0 with cluster mode the next day. It includes framework adapters for LangChain, LangGraph, and Vercel AI SDK, plus OpenTelemetry and Prometheus instrumentation at the cache layer.
The Three-Tier Architecture
Most agent caching solutions pick one layer and stop. LangChain caches LLM completions. LangGraph persists checkpoints. Agent-cache handles all three:
Tier 1: LLM Response Cache
Exact-match cache keyed on prompt text and model parameters. If your agent calls gpt-4o with identical input twice, the second call returns from Valkey in under 1ms instead of hitting the API. This is the biggest token saver when agents retry or loop over similar prompts.
Tier 2: Tool Output Cache
Caches function call results keyed on tool name and arguments. If get_weather("Sofia") runs twice with the same parameters, the cached result comes back instantly. Useful when agents re-invoke tools during backtracking or multi-step reasoning.
Tier 3: Session State Cache
Stores agent checkpoints, user intent, and execution state with per-field TTL. This is where LangGraph checkpoints live, along with any custom state your orchestrator needs to resume mid-flow.
Each tier uses a different TTL strategy. LLM responses might cache for hours if the model and prompt are stable. Tool outputs expire faster when external data changes frequently. Session state persists only as long as the user session is active.
Cache Key Design and Invalidation
The cache key for LLM responses combines prompt hash, model name, temperature, and top-p. This means changing any parameter busts the cache. If you tweak the system prompt or adjust temperature, the agent hits the API again.
Tool output keys include the function name and a hash of the arguments object. This works for deterministic tools (database lookups, API calls with stable responses) but breaks down when tools mutate external state. If your agent calls create_ticket(title, description), caching the result means subsequent calls with the same arguments return the old ticket ID instead of creating a new one.
The invalidation strategy is manual. Agent-cache does not track dependencies between cache entries. If a tool mutates state that affects future LLM calls, you must explicitly invalidate the relevant keys. The library exposes a cache.invalidate(pattern) method that accepts Redis glob patterns, but you need to know which keys to target.
Failure Modes and Degradation
When Valkey or Redis becomes unavailable, the agent has three options:
- Fail fast: Throw an error and halt execution. This makes sense if session state is critical and losing it breaks the agent’s ability to resume.
- Degrade gracefully: Skip the cache and hit the LLM or tool directly. Latency increases, token costs spike, but the agent keeps running.
- Use a local fallback: Keep an in-memory LRU cache for the current request. Session state is lost across requests, but single-request loops still benefit from caching.
Agent-cache defaults to option 2 (graceful degradation) but lets you configure the behavior per tier. You can fail fast on session state loss while degrading gracefully for LLM and tool caches.
The observability layer helps here. OpenTelemetry spans track cache hits, misses, and errors. Prometheus metrics expose hit rate, latency, and connection pool health. If your cache hit rate drops suddenly, you know to check Redis availability or inspect your invalidation logic.
Framework Adapters and Integration Points
The library ships adapters for three frameworks:
LangChain: Wraps the BaseLLMCache interface. Drop it into your chain config and LLM calls automatically cache.
LangGraph: Implements the BaseCheckpointSaver interface. Checkpoints persist to Valkey instead of requiring Redis 8 with modules.
Vercel AI SDK: Hooks into the streamText and generateText APIs. Streaming support is on the roadmap but not yet shipped.
Each adapter handles serialization differently. LangChain uses JSON for LLM responses. LangGraph uses MessagePack for checkpoints to save space. Tool outputs serialize as JSON by default but accept custom serializers if you need binary formats.
Deployment Shape
Agent-cache assumes you already run Valkey or Redis in production. It does not bundle a server or manage deployment. You point it at an existing instance (standalone, sentinel, or cluster) and it handles connection pooling.
Cluster mode support landed in v0.2.0. The library uses hash tags to ensure related keys (LLM response + tool outputs for the same agent run) land on the same shard. This avoids cross-shard transactions and keeps latency predictable.
For high-availability setups, use Redis Sentinel or Valkey’s built-in replication. The client automatically fails over to a replica if the primary goes down. Session state might lag by a few seconds during failover, but LLM and tool caches remain available.
Trade-Offs and Risks
| Concern | Risk | Mitigation |
|---|---|---|
| Stale tool outputs | Cached results don’t reflect external state changes | Use short TTLs or invalidate on mutation |
| Cache key collisions | Different prompts hash to the same key | Include model params and full prompt in key |
| Memory pressure | Large LLM responses fill Redis memory | Set max memory policy to allkeys-lru |
| Session state loss | Redis restart wipes active sessions | Persist RDB snapshots or use AOF |
| Observability overhead | Tracing every cache hit adds latency | Sample traces at 1% in production |
The biggest risk is treating the cache as a source of truth. If your agent relies on cached tool outputs to make decisions, and those outputs are stale, the agent acts on outdated information. This is fine for read-only tools (weather lookups, documentation search) but dangerous for tools that mutate state (database writes, API calls with side effects).
Code Example: LangChain Integration
import { AgentCache } from '@betterdb/agent-cache';
import { ChatOpenAI } from 'langchain/chat_models/openai';
import { initializeAgentExecutorWithOptions } from 'langchain/agents';
import { Calculator } from 'langchain/tools/calculator';
const cache = new AgentCache({
redis: { host: 'localhost', port: 6379 },
ttl: {
llm: 3600, // 1 hour for LLM responses
tool: 300, // 5 minutes for tool outputs
session: 1800 // 30 minutes for session state
},
telemetry: {
otel: true,
prometheus: { port: 9090 }
}
});
const model = new ChatOpenAI({
modelName: 'gpt-4o',
cache: cache.llmCache() // Plug into LangChain's cache interface
});
const tools = [new Calculator()];
const executor = await initializeAgentExecutorWithOptions(
tools,
model,
{ agentType: 'openai-functions' }
);
// First call hits OpenAI API
const result1 = await executor.call({
input: 'What is the weather in Sofia?'
});
// Second identical call returns from Valkey in <1ms
const result2 = await executor.call({
input: 'What is the weather in Sofia?'
});The cache.llmCache() method returns an object that implements LangChain’s BaseLLMCache interface. LangChain automatically checks the cache before calling the model and stores responses after successful completions.
Technical Verdict
Use agent-cache when:
- Your agent loops over similar prompts or re-invokes tools with identical arguments
- Token costs are a line item you need to control
- You already run Valkey or Redis in production
- You need observability into cache behavior (hit rate, latency, errors)
- You want session state to survive across requests without rebuilding from scratch
Avoid it when:
- Your tools mutate external state and caching results would cause incorrect behavior
- Your agent prompts are highly dynamic and cache hit rate would be near zero
- You don’t have infrastructure to run and monitor Redis/Valkey
- You need semantic similarity matching instead of exact-match caching (use a vector store instead)
- Your agent runs once per user and session state doesn’t matter
The library fills a gap between framework-specific caching (LangChain’s LLM cache, LangGraph’s checkpoint store) and general-purpose Redis usage. It works best when you control the agent loop and can reason about which operations are safe to cache. If your agent is a black box or you don’t understand when tools get called, start with observability and measure cache hit rate before committing to this architecture.