Harvey just open-sourced their internal legal agent benchmark: 1,671 tasks spanning 24+ practice areas, an execution harness with model adapters, and a scoring pipeline that uses all-pass rubrics instead of partial credit. The repository (780 stars, rank 7 trending in Python) is a rare public artifact showing how a production legal AI company evaluates agents against realistic professional work.
Most agent benchmarks test code generation or web navigation. Harvey LAB tests whether an agent can review an M&A data room, draft a memo on regulatory compliance, or compare contract clauses across multiple documents. The difference is not just domain vocabulary. It is the shape of the evaluation infrastructure.
Why Legal Work Breaks Toy Benchmarks
Legal tasks are multi-document, multi-step, and subjective. A single assignment might require:
- Reading 15 contracts to extract termination clauses
- Comparing those clauses against a regulatory framework
- Drafting a summary memo with citations
- Formatting the output according to firm style guidelines
There is no single correct answer. There is no unit test. The rubric might say “identifies all material risks” or “cites relevant case law.” Scoring requires an LLM judge that understands legal reasoning, not string matching.
Harvey’s benchmark exposes three infrastructure challenges that do not appear in HumanEval or SWE-bench:
- Task isolation: Each run must start with a clean document set and no leaked state from prior runs.
- Rubric scoring: Every criterion must pass. One missed clause fails the entire task.
- Model adapter abstraction: The harness must support OpenAI, Anthropic, and custom endpoints without rewriting tool schemas.
Architecture: Tasks, Harness, Adapters, Sweeps
The repository splits into four layers:
| Layer | Responsibility | Key Files |
|---|---|---|
| Task schema | Instructions, documents, rubrics, expected outputs | tasks/ directory, JSON schema |
| Execution harness | Agent loop, tool calls, document retrieval, output capture | harness/runner.py |
| Model adapters | Abstract LLM APIs (OpenAI, Anthropic, custom) | harness/adapters/ |
| Scoring pipeline | LLM judge evaluation, all-pass rubric logic, report generation | harness/scoring.py |
Task Schema
Each task is a JSON file with:
{
"task_id": "ma_dataroom_001",
"practice_area": "mergers_acquisitions",
"instructions": "Review the data room and identify all material regulatory risks...",
"documents": [
{"id": "doc_1", "path": "contracts/purchase_agreement.pdf"},
{"id": "doc_2", "path": "filings/sec_10k.pdf"}
],
"rubric": [
{"criterion": "Identifies CFIUS filing requirement", "weight": 1.0},
{"criterion": "Cites relevant Hart-Scott-Rodino thresholds", "weight": 1.0}
],
"expected_output_format": "markdown_memo"
}The schema enforces structure but allows subjective rubrics. The harness does not care whether the agent uses RAG, summarization, or chain-of-thought. It only cares that the output satisfies every rubric criterion.
Execution Harness
The harness runs a loop:
- Load task and documents into isolated context
- Send instructions to agent via model adapter
- Capture tool calls (document retrieval, web search, calculation)
- Stream agent output to buffer
- Pass output and rubric to LLM judge
- Record pass/fail per criterion
Tool calls are logged but not validated in real time. The harness trusts the agent to use tools correctly. If the agent hallucinates a document ID, the retrieval tool returns empty and the judge penalizes the output.
Model Adapters
Adapters normalize API differences:
class ModelAdapter(ABC):
@abstractmethod
def complete(self, messages: list[dict], tools: list[dict]) -> dict:
pass
class OpenAIAdapter(ModelAdapter):
def complete(self, messages, tools):
response = openai.ChatCompletion.create(
model=self.model_name,
messages=messages,
tools=tools
)
return self._normalize_response(response)
class AnthropicAdapter(ModelAdapter):
def complete(self, messages, tools):
# Convert OpenAI tool schema to Anthropic format
anthropic_tools = self._convert_tools(tools)
response = anthropic.messages.create(
model=self.model_name,
messages=messages,
tools=anthropic_tools
)
return self._normalize_response(response)The adapter layer hides provider quirks. OpenAI uses function_call, Anthropic uses tool_use blocks, and custom endpoints might use JSON-RPC. The harness sees a uniform complete() interface.
All-Pass Rubric Scoring
Harvey uses all-pass scoring: every rubric criterion must pass or the task fails. This mirrors how legal work is evaluated. A memo that identifies 9 out of 10 material risks is not 90% correct. It is wrong.
The LLM judge receives:
- Agent output
- Rubric criterion
- Ground truth examples (optional)
It returns pass or fail with reasoning. The harness aggregates results:
def score_task(output: str, rubric: list[dict]) -> dict:
results = []
for criterion in rubric:
judgment = llm_judge.evaluate(
output=output,
criterion=criterion["criterion"],
examples=criterion.get("examples")
)
results.append({
"criterion": criterion["criterion"],
"passed": judgment["passed"],
"reasoning": judgment["reasoning"]
})
overall_pass = all(r["passed"] for r in results)
return {"passed": overall_pass, "details": results}If any criterion fails, the task fails. This is stricter than weighted scoring but reflects professional standards.
Sweep Infrastructure for Reproducible Comparisons
The repository includes sweep tooling to run the same task set across multiple model configurations:
python -m harness.sweep \
--tasks tasks/ma_dataroom/ \
--models gpt-4,claude-3-opus,custom-endpoint \
--runs 3 \
--output sweeps/ma_comparison/Each sweep produces:
- Per-task pass/fail results
- Aggregated accuracy by practice area
- Tool usage statistics
- Latency and token cost breakdowns
The sweep runner isolates each model run in a separate process to prevent memory leaks or state contamination. Results are written to JSON and rendered in a dashboard (built with Plotly or similar).
Failure Modes and Observability Gaps
The benchmark exposes three common agent failure modes:
- Document retrieval errors: Agent requests a document that does not exist or misinterprets the retrieval tool schema.
- Incomplete reasoning: Agent skips a rubric criterion because it did not understand the instruction.
- Format violations: Agent returns plain text when the rubric expects structured JSON or Markdown.
The harness logs tool calls but does not trace internal reasoning steps. If an agent uses chain-of-thought, those intermediate outputs are invisible unless the model adapter explicitly captures them. This is a deliberate trade-off: Harvey prioritizes final output quality over interpretability.
Observability improvements could include:
- Trace export to OpenTelemetry or LangSmith
- Intermediate reasoning capture via structured logging
- Tool call validation before execution (fail fast on malformed requests)
None of these are implemented in the current release.
When to Use This Benchmark
Harvey LAB is useful when:
- You are building agents for professional domains (legal, finance, healthcare) where partial credit does not apply.
- You need reproducible evaluation across multiple LLM providers.
- You want to test multi-document reasoning and citation accuracy.
- You are willing to run an LLM judge for scoring (no ground truth labels).
Avoid it when:
- You need deterministic, unit-testable benchmarks (use HumanEval or MBPP).
- Your tasks have objective answers (use exact match or BLEU scoring).
- You cannot afford LLM judge costs (each task requires multiple judge calls).
- You need real-time agent evaluation (the harness is batch-oriented).
The all-pass rubric methodology is harsh. Expect low pass rates (20-40%) even with frontier models. This is intentional. Harvey is not measuring general intelligence. They are measuring readiness for professional work.
Technical Verdict
Harvey LAB is the first open-source benchmark that treats agent evaluation like a production deployment problem. The task schema, execution harness, and model adapters are production-grade. The all-pass rubric scoring is unforgiving but realistic.
Use this if you are building agents for domains where “mostly correct” is not acceptable. The infrastructure is reusable: swap the legal tasks for financial analysis, medical chart review, or engineering design review. The harness and scoring pipeline remain the same.
The main limitation is observability. The harness does not expose intermediate reasoning, tool call validation, or trace export. You will need to add instrumentation if you want to debug why an agent failed a rubric criterion.
If you are evaluating agents on toy benchmarks and wondering why they fail in production, this repository shows the gap. Professional work requires multi-document reasoning, subjective scoring, and task isolation. Harvey built the plumbing. Now you can use it.