Black-Box Testing with AI Agents: Test the Rules, Not the Code

The most valuable test report you can generate from an AI agent is a list of business rules your code violates. Not a green suite. Not coverage metrics. A plain list of places where the running system does something different from what the specification says it should do.

This happens when you keep the agent away from your source code.

Traditional test generation fails because agents use the implementation as the oracle. Point an LLM at RefundService.java and ask for tests, and it will read the code, understand what it does, then write assertions that confirm it does exactly that. If the code has a bug (refund window checks daysSinceDelivery < 30 when the policy says “within 30 days”), the agent writes a test that locks in the bug. Green suite, wrong behavior.

Black-box testing with agents flips this. You give the agent business rules, a running environment, and nothing else. No repository access. No schema. The agent generates test cases from the rules, executes them against the live system, and reports every discrepancy.

Why Code Access Weakens Test Quality

When an agent can read implementation details, it anchors on what the code does rather than what it should do. This is the test oracle problem: if your source of truth is the code under test, you cannot detect implementation drift.

The oracle contamination pattern:

1. Agent reads calculateRefund() method
2. Sees business logic: if (days < 30 && reason != "damaged")
3. Generates test: assertRefundDenied(day=30, reason="changed_mind")
4. Test passes, bug survives

The agent wrote a regression test, not a validation test. It confirmed the code’s current behavior instead of checking compliance with the rule: “Customers may return items within 30 days for any reason.”

Black-box agents avoid this by never seeing the implementation. They only know the rule. When they test day 30, they expect a refund. When the system denies it, they report a violation.

Architecture: Rules to Findings Pipeline

A black-box testing agent needs four components:

1. Rule corpus
Business requirements in structured format. Not user stories. Not Jira tickets. Testable statements like “Refund requests submitted within 30 days of delivery must be approved within 2 business days.”

2. Test case generator
Agent that reads rules and produces executable test scenarios. Each scenario includes setup state, action, and expected outcome derived purely from the rule.

3. Execution harness
Orchestrator that runs test cases against the live system (or staging environment) via API, UI automation, or message queue. Captures actual outcomes.

4. Findings reporter
Diff engine that compares expected outcomes (from rules) to actual outcomes (from execution), then formats violations as a prioritized list.

Pipeline flow:

Business Rules (markdown/YAML)
    ↓
Test Generator Agent
    ↓
Test Scenarios (JSON)
    ↓
Execution Harness (Playwright/API client)
    ↓
Actual Outcomes (JSON)
    ↓
Findings Reporter Agent
    ↓
Violation Report (markdown)

The key constraint: no component has read access to application source code or internal documentation.

Implementation: File Layout and Prompts

Directory structure:

black-box-tests/
├── rules/
│   ├── refunds.md
│   ├── pricing.md
│   └── shipping.md
├── scenarios/
│   └── generated/
│       ├── refunds_001.json
│       └── refunds_002.json
├── results/
│   └── 2026-10-26_run.json
├── findings/
│   └── 2026-10-26_violations.md
└── orchestrator.py

Rule format (rules/refunds.md):

# Refund Policy Rules

## R-001: Refund Window
Customers may request a refund within 30 days of delivery.
- Delivery date is the date the carrier marks the package as delivered.
- Day 1 is the day after delivery.
- Day 30 is included in the refund window.

## R-002: Refund Processing Time
Approved refund requests must be processed within 2 business days.
- Business days exclude weekends and US federal holidays.
- Processing means funds returned to original payment method.

Test generator prompt:

You are a test case generator. Read the business rule provided below.
Generate 5 test scenarios that validate compliance with this rule.

For each scenario, output JSON with:
- scenario_id: unique identifier
- rule_id: which rule this tests
- setup: preconditions (user state, system state)
- action: what the user does
- expected_outcome: what should happen per the rule
- edge_case: boolean, true if this tests a boundary

Rule:
{rule_text}

Output only valid JSON array. No explanation.

Generated scenario (scenarios/generated/refunds_001.json):

{
  "scenario_id": "refunds_001_day30",
  "rule_id": "R-001",
  "setup": {
    "user_id": "test_user_001",
    "order_id": "ORD-12345",
    "delivery_date": "2026-09-26",
    "current_date": "2026-10-26"
  },
  "action": {
    "type": "api_call",
    "endpoint": "/api/refunds",
    "method": "POST",
    "body": {
      "order_id": "ORD-12345",
      "reason": "changed_mind"
    }
  },
  "expected_outcome": {
    "status": "approved",
    "reason": "Request is on day 30, which is within the 30-day window per R-001"
  },
  "edge_case": true
}

Execution harness (orchestrator.py snippet):

def execute_scenario(scenario):
    # Setup: create test data
    setup_test_user(scenario['setup'])
    
    # Execute action
    if scenario['action']['type'] == 'api_call':
        response = requests.request(
            method=scenario['action']['method'],
            url=BASE_URL + scenario['action']['endpoint'],
            json=scenario['action']['body'],
            headers=get_auth_headers(scenario['setup']['user_id'])
        )
        
        actual_outcome = {
            'status_code': response.status_code,
            'body': response.json(),
            'timestamp': datetime.utcnow().isoformat()
        }
    
    return {
        'scenario_id': scenario['scenario_id'],
        'expected': scenario['expected_outcome'],
        'actual': actual_outcome
    }

Findings reporter prompt:

You are a compliance auditor. Compare the expected outcome (from business rules)
to the actual outcome (from system execution).

If they differ, write a violation report entry with:
- Rule violated
- What should have happened
- What actually happened
- Severity (critical, high, medium, low)

Expected:
{expected_json}

Actual:
{actual_json}

Output markdown. If no violation, output "PASS".

Sample findings report (findings/2026-10-26_violations.md):

# Compliance Violations: 2026-10-26

## Critical Violations

### V-001: Refund Window Boundary Error
- **Rule:** R-001 (Refund Window)
- **Scenario:** refunds_001_day30
- **Expected:** Refund approved (day 30 is within window)
- **Actual:** Refund denied with error "Request outside 30-day window"
- **Root Cause:** Implementation uses `<` instead of `<=` for day comparison
- **Impact:** Customers denied valid refunds on final eligible day

## High Violations

### V-002: Processing Time SLA Breach
- **Rule:** R-002 (Refund Processing Time)
- **Scenario:** refunds_005_processing
- **Expected:** Refund processed within 2 business days
- **Actual:** Refund processed after 4 business days
- **Impact:** SLA violation, potential customer complaints

Observability Without Internal State

Black-box testing requires external observability primitives because you cannot instrument the application:

API response inspection:
Capture status codes, response bodies, headers. Compare to rule-derived expectations.

State verification queries:
Use read-only API endpoints to check side effects. After submitting a refund request, call GET /api/refunds/{id} to verify status changed.

Event stream monitoring:
If the system publishes domain events, subscribe to the stream. Verify events match rule expectations (refund approved event within 2 business days).

Database snapshots (when available):
If you have read-only DB access, query final state. Still black-box because you are not reading application code, just verifying data outcomes.

Time-based assertions:
For SLA rules, record timestamps at each step. Calculate elapsed time. Compare to rule thresholds.

The constraint is you cannot trace execution paths or inspect internal variables. You only observe inputs, outputs, and externally visible state changes.

Handling Non-Deterministic Execution Paths

Agents may generate different test cases on each run. The execution path through your system may vary based on timing, load, or randomness in the agent’s tool selection.

Versioning test scenarios:
Commit generated scenarios to git before execution. Tag with generation timestamp. This lets you replay exact scenarios later.

Idempotent setup:
Each scenario must create its own isolated test data. Use unique user IDs, order IDs, timestamps. Tear down after execution.

Retry logic for flaky outcomes:
If a test fails due to timing (e.g., async processing not complete), retry with exponential backoff. Mark as flaky if it passes on retry.

Scenario deduplication:
Before executing, hash scenario inputs. Skip duplicates. This prevents redundant API calls when the agent generates similar test cases.

Execution replay:
Store request/response pairs. When a violation is reported, you can replay the exact HTTP exchange without re-running the agent.

Trade-offs and Failure Modes

Aspect	Black-Box Agent Testing	Traditional Unit Tests
Oracle source	Business rules (external)	Code behavior (internal)
Detects drift	Yes, reports rule violations	No, locks in current behavior
Execution speed	Slow (full system calls)	Fast (isolated functions)
Setup complexity	High (needs running environment)	Low (mock dependencies)
Coverage metric	Rule coverage	Code coverage
Debugging	Hard (no stack traces)	Easy (direct code access)
Maintenance	Low (rules change slowly)	High (refactors break tests)

When black-box testing fails:

Performance testing:
Agents cannot validate latency or throughput requirements without internal metrics. You need instrumentation.

Security boundaries:
Testing authentication logic requires knowing internal token validation. Black-box can only test outcomes (401 vs 200), not mechanisms.

Complex state machines:
If a rule depends on 15 prior state transitions, setting up the scenario becomes brittle. You need access to state manipulation tools.

Algorithmic correctness:
Rules like “recommendation engine must use collaborative filtering” cannot be validated externally. You must inspect the algorithm.

Race conditions:
Timing bugs require controlled execution, not external observation. You need to pause threads or inject delays.

Technical Verdict

Use black-box agent testing when:

• You have well-defined business rules separate from code
• Implementation details change frequently (refactors, rewrites)
• Compliance or audit requirements demand rule validation
• Your team struggles with test maintenance burden
• The system has stable external APIs or UI

Avoid when:

• Rules are vague or change constantly
• You need to test internal algorithms or data structures
• Performance or concurrency bugs are the primary concern
• The system has no external interface (library, batch job)
• Setup cost (running environment, test data) exceeds value

The real power is not replacing unit tests. It is catching the gap between what you built and what you were supposed to build. That gap is invisible to tests that read the code.

Source Links

• Black-Box Testing with AI Agents: Test the Rules, Not the Code (Dev.to)