Google Chrome, Microsoft Edge, Mozilla Firefox, and MDN just shipped a shared eval framework to measure whether coding agents actually use modern web APIs or fall back to legacy patterns. The project hit 2,873 GitHub stars and trended #1 for TypeScript because it addresses a core problem: LLM training data is polluted with legacy code, so agents default to jQuery animations instead of View Transitions, even when the modern API is better supported.
Modern Web Guidance is not a linter. It is a continuous testing pipeline that measures agent behavior at the tool level, then publishes results to a public dashboard. The plumbing reveals how browser vendors coordinate on a shared skill format, how static analysis works as a pre-flight check, and what happens when you try to change agent behavior without retraining the model.
The Problem: Training Data Pollution
LLMs learn from vast amounts of legacy code. When you ask an agent to animate a modal backdrop, it often generates jQuery or manual DOM manipulation because that pattern appears more frequently in training data than the View Transitions API. The agent knows the modern API exists, but it does not know when to prefer it.
This is not a knowledge problem. It is a retrieval and ranking problem. The agent needs:
- A way to search for relevant modern patterns at inference time
- A structured skill format that works across Claude, Copilot, and Cursor
- A feedback loop that measures whether it actually chose the modern API
Modern Web Guidance solves this by shipping three CLI commands and an eval dashboard.
CLI Commands: Search, Retrieve, Legacy Scan
The project ships as an npm package with three commands:
# Search for relevant guides
npx modern-web-guidance@latest search "animate a dialog modal backdrop"
# Retrieve a guide by ID
npx modern-web-guidance@latest retrieve "animate-to-from-top-layer"
# Scan a codebase for legacy patterns
npx modern-web-guidance@latest legacy-scan .The search command returns a ranked list of guides. Each guide includes:
- A unique ID
- A description of the modern pattern
- Browser compatibility data
- Code examples
- Links to MDN and spec documents
The retrieve command fetches the full guide by ID. This is the format agents consume. It is structured as a skill: a task description, a solution, and a set of constraints.
The legacy-scan command is the pre-flight check. It runs static analysis on your codebase and flags patterns that have modern equivalents. This is not a blocking step. It is a signal to the agent that legacy patterns exist and should be replaced.
Eval Dashboard: Measuring Agent Behavior
The eval dashboard at googlechrome.github.io/modern-web-guidance-src/ shows whether agents actually adopt modern APIs. The dashboard tracks:
- Which guides agents retrieve
- Which code patterns agents generate
- Whether the generated code matches the modern pattern or falls back to legacy
The eval pipeline works like this:
- Prompt generation: The system generates prompts that require specific web APIs (e.g., “animate a modal backdrop”)
- Agent execution: The agent runs with access to Modern Web Guidance skills
- Static analysis: The output is scanned for modern vs. legacy patterns
- Scoring: Each result is scored as modern, legacy, or mixed
- Dashboard update: Results are published to the public dashboard
This is continuous testing. The pipeline runs on every commit to the guidance repository, so the dashboard reflects the current state of agent behavior.
Skill Format: Cross-Agent Compatibility
The skill format is designed to work across Claude, Copilot, Cursor, and any other agent that can consume structured context. Each skill is a Markdown file with frontmatter:
---
id: animate-to-from-top-layer
title: Animate to/from the top layer
tags: [animation, dialog, popover]
compatibility:
chrome: 117
edge: 117
firefox: 129
safari: 18
---
## Task
Animate a dialog or popover when it opens or closes, including transitions to and from the top layer.
## Solution
Use the View Transitions API with `@starting-style` and `overlay` pseudo-element.
## Constraints
- Do not use jQuery or manual DOM manipulation
- Do not use `requestAnimationFrame` for this use case
- Ensure the animation works when the element enters or exits the top layerThe frontmatter includes browser compatibility data from caniuse.com. This lets agents make informed decisions about fallbacks. If the user’s target browser does not support View Transitions, the agent can fall back to a legacy pattern, but it does so explicitly.
Static Analysis as a Guardrail
The legacy-scan command runs static analysis to detect patterns that have modern equivalents. It looks for:
- jQuery usage when native DOM APIs are sufficient
XMLHttpRequestwhenfetchis availablerequestAnimationFrameloops when CSS animations or View Transitions are better- Polyfills for APIs that are now widely supported
The scan does not block execution. It generates a report that the agent can use to prioritize refactoring. This is a guardrail, not a gate.
The scan also feeds the eval pipeline. When the pipeline generates a prompt, it runs the scan on the agent’s output. If the output contains legacy patterns, the eval is marked as a failure.
Architecture: How the Eval Pipeline Works
The eval pipeline is a GitHub Actions workflow that runs on every commit. Here is the flow:
- Prompt generation: A test harness generates prompts from a template library
- Agent invocation: The harness invokes the agent with Modern Web Guidance skills in context
- Output capture: The agent’s generated code is captured as a file
- Static analysis: The
legacy-scancommand runs on the output - Scoring: The output is scored based on whether it uses modern or legacy patterns
- Dashboard update: The results are committed to the
gh-pagesbranch
The dashboard is a static site built with Vite. It reads the eval results from JSON files and renders them as tables and charts. The dashboard shows:
- Pass rate over time
- Which guides are most frequently retrieved
- Which legacy patterns are most common in agent output
Coordination: How Browser Vendors Agree on Skills
Modern Web Guidance is stewarded by Google Chrome, Microsoft Edge, Mozilla Firefox, and MDN. This is unusual. Browser vendors rarely coordinate on tooling, especially tooling for agents.
The coordination works because the skill format is vendor-neutral. Each skill is a Markdown file with frontmatter. There is no vendor-specific API. The skills are published to a shared repository, and each vendor can consume them however they want.
The governance model is also lightweight. The project uses a GOVERNANCE.md file that defines:
- Who can approve new skills
- How conflicts are resolved
- How the eval pipeline is maintained
The key insight is that the skills are not normative. They are guidance. If a vendor disagrees with a skill, they can fork the repository and maintain their own version. In practice, this has not happened because the skills are based on web standards, not vendor preferences.
Trade-offs: When Evals Change Agent Behavior
The eval framework measures whether agents adopt modern APIs, but it does not guarantee they will. Here are the trade-offs:
| Approach | Pros | Cons |
|---|---|---|
| Eval-only | Measures current behavior, no agent changes required | Does not change behavior, only observes it |
| Eval + skills in context | Agents can retrieve modern patterns at inference time | Agents may ignore skills if training data is stronger |
| Eval + fine-tuning | Changes model weights to prefer modern patterns | Expensive, requires retraining, vendor-specific |
| Eval + static analysis guardrails | Blocks legacy patterns before execution | May block valid use cases, requires manual overrides |
Modern Web Guidance uses the second approach: eval + skills in context. This is the cheapest way to change agent behavior without retraining the model. The risk is that agents may ignore the skills if the training data signal is stronger.
The eval dashboard makes this visible. If the pass rate is low, it means the skills are not working. The team can then adjust the skill format, add more examples, or escalate to the agent vendor.
Failure Modes: What Breaks
The eval framework has several failure modes:
- Static analysis false positives: The
legacy-scancommand may flag modern code as legacy if it uses a pattern that looks similar - Skill retrieval failures: Agents may fail to retrieve the right skill if the search query is ambiguous
- Agent ignores skills: Agents may generate legacy code even when the skill is in context
- Browser compatibility drift: The compatibility data may become stale if browser vendors ship new features
The first two are engineering problems. The team can improve the static analysis rules and the search ranking algorithm.
The third is a model problem. If the agent consistently ignores skills, it means the training data signal is too strong. The only fix is to retrain the model or add stronger guardrails.
The fourth is a data freshness problem. The team solves this by pulling compatibility data from caniuse.com on every build. If the data is stale, the build fails.
Technical Verdict
Use Modern Web Guidance if you are building or operating coding agents that generate web code. The eval framework gives you a baseline to measure whether your agent is adopting modern APIs or falling back to legacy patterns. The skill format is vendor-neutral, so you can use it with Claude, Copilot, Cursor, or any other agent.
Avoid it if you are building agents for non-web domains. The skills are specific to web APIs, and the static analysis rules are tuned for JavaScript and CSS. The framework is not generalizable to other languages or platforms.
The project is most useful as a reference architecture. If you are building evals for agent tool adoption, the pipeline shows how to combine static analysis, skill retrieval, and continuous testing. The dashboard shows how to make eval results visible to both engineers and agent vendors.
The key insight is that evals can change agent behavior without retraining the model, but only if the skills are in context and the agent is incentivized to use them. Modern Web Guidance does not solve the incentive problem. It just makes the behavior visible.