mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

Dev Tools

Deterministic PR Gates for Agent-Generated Code: How to Reject Fixes That Can't Prove Themselves

Build a Python gate that validates agent PRs with evidence, not promises. No LLM grades the LLM. Checks test coverage, regressions, and proof of fix.

Source: dev.to
Deterministic PR Gates for Agent-Generated Code: How to Reject Fixes That Can't Prove Themselves

AI agents now open 17 million pull requests per month. Most of them claim to fix bugs. Some of them do. Many look plausible but break existing tests, weaken coverage, or solve the wrong problem. The gap between “I fixed it” and “here is proof” is where human review time burns.

Sarvar Nadaf built a deterministic Python gate that sits between agent commit and human review. It checks evidence, not promises. No LLM grades the LLM. The gate runs against Amazon Nova Pro agent output and posts rejection reasons directly to the PR when proof is missing.

The Problem: Claims Without Evidence

An agent reads a GitHub issue, generates a fix, opens a PR, and writes “done, one fix.” A human reviewer sees green CI and assumes the agent did what it said. Then production breaks because the agent deleted a test to make the suite pass, or the fix only works for one edge case, or the claimed bug was never actually solved.

Traditional CI checks syntax, runs tests, and reports pass or fail. That tells you the code compiles and existing tests still work. It does not tell you:

  • Did the agent fix the specific bug in the issue?
  • Did it weaken test coverage to make the suite green?
  • Did it introduce a regression in a different code path?
  • Is there a test that proves the fix works?

The gate Nadaf built answers those questions with deterministic checks before a human looks at the diff.

Architecture: Deterministic Validation Without Model Grading

The system has three layers:

  1. Agent execution: Amazon Nova Pro runs as a Kiro Crew job, reads a GitHub issue, generates a fix, commits to a branch.
  2. Proof gate: Python script runs on the commit, compares old and new code, checks test results, calculates coverage delta.
  3. PR status check: Gate posts results as a GitHub status check with specific rejection reasons or approval.

The gate does not ask another LLM to evaluate the fix. It runs deterministic checks:

  • Test regression check: Run the test suite on the old commit, then on the new commit. If a test that passed before now fails, reject.
  • Coverage delta check: Measure line coverage before and after. If coverage drops, flag it. If the agent deleted tests, reject.
  • Fix proof check: The issue description should map to a test. Run that test on the old code (should fail), then on the new code (should pass). If both pass or both fail, the fix is unproven.
  • Scope boundary check: If the agent changed files unrelated to the issue, flag scope creep.

The gate outputs a structured rejection or approval:

# Simplified gate logic
def validate_fix(old_commit, new_commit, issue_id):
    old_tests = run_tests(old_commit)
    new_tests = run_tests(new_commit)
    
    # Check for test regression
    regressions = [t for t in old_tests.passed if t in new_tests.failed]
    if regressions:
        return reject(f"Tests now failing: {regressions}")
    
    # Check for deleted tests
    deleted = set(old_tests.all) - set(new_tests.all)
    if deleted:
        return reject(f"Tests deleted: {deleted}")
    
    # Check fix proof
    issue_test = find_test_for_issue(issue_id)
    if issue_test:
        old_result = run_single_test(old_commit, issue_test)
        new_result = run_single_test(new_commit, issue_test)
        if old_result == "pass" and new_result == "pass":
            return reject("Test passed before fix, no proof of bug")
        if old_result == "fail" and new_result == "fail":
            return reject("Test still fails, fix unproven")
    
    # Check coverage delta
    old_cov = measure_coverage(old_commit)
    new_cov = measure_coverage(new_commit)
    if new_cov < old_cov:
        return warn(f"Coverage dropped {old_cov - new_cov}%")
    
    return approve()

The gate runs in CI as a required status check. If it rejects, the PR cannot merge until a human overrides or the agent submits a new commit.

Failure Mode Taxonomy for Agent PRs

Agent-generated PRs fail in predictable patterns. The gate needs different logic for each:

Failure ModeSignalGate Response
Hallucinated fixTest for issue does not exist, or passes on both commitsReject: “No test proves this fix”
Test deletionFewer tests run on new commit than oldReject: “Tests deleted to pass suite”
Partial solutionIssue test passes, but related tests now failReject: “Fix breaks related functionality”
Scope creepFiles changed that are not mentioned in issueWarn: “Changes outside issue scope”
Coverage dropLine coverage decreasesWarn: “Coverage dropped X%“
RegressionPreviously passing test now failsReject: “Test regression detected”

The gate does not try to understand intent. It checks evidence. If the evidence is missing, the PR is rejected with a specific reason.

Deployment Shape

The system runs as a GitHub Action triggered on PR open or commit push:

  1. Action checks out both the base branch and the PR branch.
  2. Runs the test suite on the base branch, captures results and coverage.
  3. Runs the test suite on the PR branch, captures results and coverage.
  4. Compares results using the deterministic gate logic.
  5. Posts a status check to the PR with approval or rejection reasons.

The gate does not block CI. It runs in parallel with other checks. But it is marked as a required status check, so the PR cannot merge until it passes.

For the agent execution layer, Kiro Crew orchestrates the Nova Pro agent:

  • Reads the GitHub issue via API.
  • Generates a fix using Nova Pro with the issue description and relevant code context.
  • Commits the fix to a new branch.
  • Opens a PR with a description that includes the issue ID.

The gate reads the issue ID from the PR description and uses it to find the relevant test.

Observability and Debugging

The gate logs every decision with evidence:

  • Which tests passed or failed on each commit.
  • Coverage percentages before and after.
  • Which files changed and whether they map to the issue.
  • The specific test that should prove the fix, and its results on both commits.

Logs go to CloudWatch (or equivalent) and are also posted as a PR comment. This gives the human reviewer context if they decide to override the gate.

The gate also tracks false positives: cases where it rejected a valid fix. These are flagged for review so the gate logic can be tuned.

Security Boundaries

The gate runs in a sandboxed CI environment. The agent does not have write access to the main branch. It can only open PRs. The gate runs with read-only access to the repo and write access to PR status checks.

The agent’s GitHub token is scoped to:

  • Read issues.
  • Create branches.
  • Open PRs.
  • Read code.

It cannot merge, delete branches, or modify CI configuration.

The gate’s token is scoped to:

  • Read PR metadata.
  • Run tests.
  • Post status checks.

It cannot modify code or merge PRs.

This keeps the blast radius small. If the agent or gate is compromised, the attacker can spam PRs but cannot push to main.

Trade-Offs and Limitations

What this solves:

  • Filters out agent PRs that cannot prove they fixed the bug.
  • Catches test deletions and coverage drops before human review.
  • Reduces human review time by rejecting obvious failures early.

What this does not solve:

  • Logic bugs that pass all tests (the test suite itself may be incomplete).
  • Fixes that work but are inefficient or poorly structured.
  • Issues where the correct fix is ambiguous and requires human judgment.

Cost:

  • Adds 30-60 seconds to PR checks (time to run tests twice).
  • Requires maintaining the gate logic as the codebase evolves.
  • False positives require human override, which adds friction.

When the gate fails:

  • If the test suite is flaky, the gate will reject valid fixes.
  • If the issue description does not map to a specific test, the gate cannot verify the fix.
  • If the agent fixes a bug by refactoring, the gate may flag scope creep even if the refactor is necessary.

Technical Verdict

Use this pattern when:

  • You have an agent that opens PRs automatically.
  • Your test suite is stable and covers the bugs you care about.
  • You want to reduce human review time by filtering out provably bad fixes.
  • You can tolerate some false positives in exchange for catching most bad PRs.

Avoid this pattern when:

  • Your test suite is flaky or incomplete (the gate will reject too many valid fixes).
  • The bugs you care about are not covered by tests (the gate cannot verify fixes).
  • You need the agent to do exploratory refactoring (the gate will flag scope creep).
  • You cannot afford the extra 30-60 seconds per PR check.

The key insight is that deterministic checks can catch most bad agent PRs without needing another LLM to grade the output. The gate does not understand code. It checks evidence. If the evidence is missing, the PR is rejected. That is enough to save human reviewers from wasting time on fixes that cannot prove themselves.

Tags

agentic-ai orchestration infrastructure

Primary Source

dev.to ↗