Microsoft just released RAMPART (Risk Assessment & Measurement Platform for Agentic Red Teaming), a pytest-native framework that treats agent safety testing as a first-class CI/CD concern. Instead of running red-team exercises as separate audit steps, RAMPART lets you write adversarial tests alongside your unit and integration tests. The framework covers adversarial attacks, benign failures, and broad harm categories using evaluation-driven assertions that fit existing pytest workflows.
This matters because agents are moving from demos to production. Teams need tooling that integrates with their existing test infrastructure, not standalone security audits that happen after deployment. RAMPART is trending at #11 on GitHub for Python and ships with an OpenSSF Scorecard badge, signaling production-grade security posture.
Why Pytest for Red-Teaming
Most agent safety testing happens in notebooks or custom scripts. You run a batch of prompts, eyeball the outputs, maybe log some metrics. RAMPART takes a different approach: it structures adversarial tests as pytest fixtures and parameterized test cases. This means:
- Safety tests run in the same CI/CD pipeline as your unit tests
- You get pytest’s reporting, parallelization, and plugin ecosystem for free
- Test isolation is handled by pytest’s fixture scoping, not custom teardown logic
- Assertions use familiar pytest syntax, not domain-specific evaluation DSLs
The framework targets agentic AI applications specifically, not general ML models. It assumes your system has tool calls, state mutations, and side effects (API calls, file writes, database updates). RAMPART provides primitives for testing these behaviors under adversarial conditions.
Architecture: Fixtures, Evaluators, and Harm Categories
RAMPART’s core abstraction is the attack fixture. The framework parameterizes these fixtures across harm categories:
- Hate speech and harassment
- Self-harm and violence
- Sexual content
- Privacy violations
- Misinformation and fraud
- Malware and exploitation
Each harm category maps to a set of attack templates. You can extend these templates or write custom ones. The framework also supports benign failure modes (hallucinations, tool misuse, context leakage) that aren’t adversarial but still represent safety risks.
The repository description confirms that RAMPART provides “evaluation-driven assertions” that integrate with pytest. The framework follows pytest conventions: you define test functions that exercise your agent against adversarial inputs, then assert on the outputs using evaluation logic. The exact evaluation primitives and assertion APIs are not fully documented in the available repository materials, but the architecture follows standard pytest patterns for fixture injection and parameterized testing.
This structure lets you version-control your adversarial test suite, run it on every commit, and fail builds when agents produce unsafe outputs.
Test Isolation for Stateful Agents
Agents have side effects. A single test might trigger API calls, write files, or mutate shared state. RAMPART handles isolation through pytest’s fixture scoping mechanisms.
Fixture scoping controls when agent instances are created and destroyed. Use scope="function" for full isolation (new agent per test), scope="module" for shared state across a test file, or scope="session" for expensive setup (model loading, database connections).
For agents that make real API calls, you’ll need to implement mock injection patterns. This avoids rate limits and side effects while still testing the agent’s decision-making logic. Standard pytest mocking libraries (pytest-mock, unittest.mock) work with RAMPART’s test structure.
Deployment Shape and CI/CD Integration
RAMPART runs anywhere pytest runs. The typical deployment shape is:
- Local development: Run safety tests during feature development
- Pre-commit hooks: Block commits that fail safety checks
- CI/CD pipeline: Run full adversarial test suite on every PR
- Scheduled jobs: Run extended test suites (thousands of prompts) nightly
The framework supports parameterized test generation for large-scale testing. You can define a single test function and parameterize it across adversarial datasets, scaling to thousands of test cases that run in parallel using pytest-xdist.
For observability, RAMPART integrates with pytest plugins like pytest-html (HTML reports), pytest-json-report (structured logs), and pytest-xdist (distributed execution). You can also export test results to monitoring systems (Datadog, Prometheus) using custom pytest hooks.
Potential Challenges and Failure Modes
Evaluation logic brittleness: Any evaluation approach (toxicity detection, refusal detection) relies on heuristics or models that can miss novel attack patterns. You need to version-control your evaluation logic and update it as attacks evolve.
Test suite maintenance: Adversarial datasets go stale. If you parameterize tests over a static dataset, you won’t catch new attack vectors. Plan to refresh datasets quarterly or integrate with live red-team feeds.
Side effect management: If your agent makes real API calls during tests, you risk rate limits, cost overruns, or data corruption. Use mocking aggressively, and reserve real API calls for integration tests that run less frequently.
Assertion threshold tuning: Threshold-based assertions (e.g., “toxicity score below X”) require calibration. Set thresholds too low and you get false positives. Set them too high and you miss real safety issues. Start with conservative thresholds and adjust based on false positive rates.
Parallel execution conflicts: If tests share state (database, file system, external APIs), parallel execution will cause race conditions. Use pytest’s --dist loadscope to run tests from the same module sequentially, or isolate state per test with fixtures.
Technical Verdict
Use RAMPART when:
- You already use pytest for testing and want to add safety tests to your existing pipeline
- Your agents have side effects (tool calls, state mutations) that need isolation
- You need version-controlled, reproducible adversarial test suites
- You want safety testing to fail builds, not just generate reports
Avoid RAMPART when:
- You need runtime safety guardrails (use input/output filters or sandboxing instead)
- Your agents are stateless LLM wrappers (consider Anthropic’s Evals or OpenAI’s evals library for simpler evaluation workflows)
- You require domain-specific harm taxonomies not covered by the built-in categories
- You need real-time adversarial testing in production (RAMPART is a dev-time tool)
RAMPART shifts agent safety testing from a post-hoc audit to a continuous integration concern. It won’t catch every attack, but it makes safety testing a first-class part of your development workflow. The pytest integration is the key insight: by reusing existing test infrastructure, RAMPART reduces the friction of writing and running adversarial tests.