AWS Agent Toolkit Skills: How Amazon Packages SageMaker Expertise as Executable Agent Functions

AWS just shipped the aws-ai-ml skill for their Agent Toolkit, giving coding agents like Kiro, Claude Code, and Codex deep SageMaker inference expertise. This is not a wrapper around the SageMaker API. It is a structured way to package complex platform knowledge (benchmarking, deployment comparison, optimization patterns) as installable agent functions that generate executable Python SDK v3 code.

The pattern matters because it reveals how cloud vendors are turning platform complexity into agent-first interfaces. Instead of expecting agents to learn every SageMaker deployment option, AWS ships a skill that translates natural language intent into runnable code.

What the Skill Actually Does

The aws-ai-ml skill sits between an agent’s reasoning layer and the SageMaker Python SDK. When you describe what you want (benchmark Llama 3.1 70B on different instance types, compare real-time vs. serverless inference costs), the skill generates executable code that:

• Configures SageMaker endpoints with appropriate instance types and model containers
• Runs benchmark workloads with realistic token distributions
• Collects latency, throughput, and cost metrics
• Produces comparison tables or deployment recommendations

The agent does not need to know SageMaker’s deployment taxonomy or SDK method signatures. The skill encapsulates that knowledge and emits valid code.

Skill Interface and Boundaries

AWS structures the skill as a set of callable functions with defined inputs and outputs. The agent sees function signatures like:

def benchmark_inference_deployment(
    model_id: str,
    instance_types: list[str],
    workload_profile: dict,
    duration_minutes: int
) -> dict:
    """
    Benchmark SageMaker inference across instance types.
    Returns latency percentiles, throughput, and cost per 1M tokens.
    """

The skill provides:

• Declarative knowledge: Valid instance type combinations, model container URIs, recommended configurations for common models
• Code templates: Pre-built patterns for endpoint creation, benchmark execution, metric collection
• Validation logic: Checks for invalid instance/model pairings before generating code

The agent provides:

• Intent parsing: Translating user requests into skill function calls
• Parameter selection: Choosing instance types, workload profiles, duration based on user constraints
• Result interpretation: Deciding which deployment option to recommend based on benchmark output

The boundary is clear. The skill does not reason about trade-offs. It generates correct code for the parameters the agent selects.

Code Generation Flow

When an agent invokes the skill:

1. Agent parses user intent: “Compare Llama 3.1 70B inference cost on ml.g5.12xlarge vs. ml.p4d.24xlarge for a chatbot workload”
2. Agent calls skill function: benchmark_inference_deployment(model_id="meta-llama/Llama-3.1-70B", instance_types=["ml.g5.12xlarge", "ml.p4d.24xlarge"], workload_profile="chatbot", duration_minutes=30)
3. Skill generates SageMaker SDK code:

import sagemaker
from sagemaker.huggingface import HuggingFaceModel

session = sagemaker.Session()
role = sagemaker.get_execution_role()

results = {}
for instance_type in ["ml.g5.12xlarge", "ml.p4d.24xlarge"]:
    model = HuggingFaceModel(
        model_data="s3://sagemaker-models/llama-3.1-70b",
        role=role,
        transformers_version="4.37",
        pytorch_version="2.1",
        py_version="py310",
    )
    
    predictor = model.deploy(
        initial_instance_count=1,
        instance_type=instance_type,
        endpoint_name=f"llama-70b-{instance_type.replace('.', '-')}"
    )
    
    # Run benchmark workload
    latencies = []
    for _ in range(100):
        response = predictor.predict({
            "inputs": "What is machine learning?",
            "parameters": {"max_new_tokens": 150}
        })
        latencies.append(response["latency_ms"])
    
    results[instance_type] = {
        "p50_latency": np.percentile(latencies, 50),
        "p99_latency": np.percentile(latencies, 99),
        "cost_per_hour": get_instance_cost(instance_type)
    }
    
    predictor.delete_endpoint()

4. Agent executes code: Runs in user’s AWS environment, collects metrics
5. Agent interprets results: Recommends ml.g5.12xlarge based on cost/latency trade-off

The skill’s output is always executable code, not a JSON response or API call. This keeps the agent’s execution model simple: call skill, run code, interpret output.

Versioning and SDK Drift

SageMaker’s Python SDK changes frequently. New instance types ship, model containers update, API methods deprecate. The skill must handle this without breaking agents that cached old skill definitions.

AWS likely versions skills with semantic versioning:

• Major version: Breaking changes to skill function signatures
• Minor version: New functions, new parameters with defaults
• Patch version: Bug fixes, updated instance type lists, new model containers

Agents specify which skill version to install:

skills:
  - name: aws-ai-ml
    version: "1.2.0"
    source: aws-agent-toolkit

When the SageMaker SDK deprecates a method, AWS has two options:

1. Emit compatibility shims: Skill generates code that works with both old and new SDK versions
2. Force upgrade: Skill requires minimum SDK version, fails fast if environment is outdated

The skill likely includes SDK version checks in generated code:

import sagemaker
assert sagemaker.__version__ >= "3.0.0", "Skill requires SageMaker SDK >= 3.0.0"

This pushes version management to the execution environment, not the skill definition.

Error Surface and Failure Modes

The skill can fail at multiple layers:

Failure Mode	Detection Point	Error Handling
Invalid instance/model pairing	Skill validation	Return error before code generation
Quota limits (endpoint count)	SDK execution	Raise SageMaker quota exception
Insufficient IAM permissions	SDK execution	Raise IAM permission error
Benchmark timeout	Generated code	Return partial results with timeout flag
Cost threshold exceeded	Agent logic	Agent halts execution, prompts user

The skill cannot prevent all failures because it generates code that runs in the user’s environment. It can only validate inputs it controls (instance types, model IDs, parameter ranges).

When an agent requests an invalid configuration (Llama 3.1 405B on ml.t3.medium), the skill should fail fast:

{
  "error": "InvalidConfiguration",
  "message": "Model meta-llama/Llama-3.1-405B requires minimum 8x A100 GPUs. ml.t3.medium has 0 GPUs.",
  "suggested_instances": ["ml.p4d.24xlarge", "ml.p5.48xlarge"]
}

This keeps the agent from generating code that will fail at runtime.

Observability and Debugging

When generated code fails, the agent needs visibility into what went wrong. The skill likely includes structured logging:

import logging
logger = logging.getLogger("aws-ai-ml-skill")

logger.info("Deploying endpoint", extra={
    "model_id": model_id,
    "instance_type": instance_type,
    "endpoint_name": endpoint_name
})

try:
    predictor = model.deploy(...)
    logger.info("Endpoint deployed successfully", extra={
        "endpoint_arn": predictor.endpoint_arn
    })
except Exception as e:
    logger.error("Deployment failed", extra={
        "error_type": type(e).__name__,
        "error_message": str(e)
    }, exc_info=True)
    raise

Agents can parse these logs to provide useful error messages to users. Without structured logging, the agent only sees raw SDK exceptions.

Deployment Shape

The skill itself is a Python package installed in the agent’s execution environment:

pip install aws-agent-toolkit[ai-ml]

The agent imports the skill and calls its functions:

from aws_agent_toolkit.skills.ai_ml import benchmark_inference_deployment

code = benchmark_inference_deployment(
    model_id="meta-llama/Llama-3.1-70B",
    instance_types=["ml.g5.12xlarge"],
    workload_profile="chatbot",
    duration_minutes=30
)

exec(code)  # Agent executes generated code

This keeps the skill’s dependencies isolated. If the skill requires specific versions of boto3 or sagemaker SDK, they do not pollute the agent’s environment.

Security Boundaries

The skill generates code that runs with the agent’s AWS credentials. This creates several risks:

• Runaway costs: Agent deploys expensive instances and forgets to delete endpoints
• Data exfiltration: Generated code sends model outputs to attacker-controlled S3 bucket
• Privilege escalation: Agent uses skill to create IAM roles with broader permissions

AWS likely mitigates these with:

1. IAM permission scoping: Skill-generated code only calls SageMaker APIs, not IAM or S3 write operations
2. Cost guardrails: Agent enforces spending limits before executing generated code (see AgentCore Payments pattern)
3. Code review: Agent shows generated code to user before execution in high-risk scenarios

The skill cannot enforce these boundaries itself. It only generates code. The agent or execution environment must apply controls.

Comparison: Skill vs. Direct SDK Access

Approach	Agent Complexity	Code Quality	Maintenance Burden
Direct SDK	High (agent learns all SageMaker APIs)	Variable (depends on agent training)	Low (AWS maintains SDK)
Skill-based	Low (agent calls skill functions)	High (skill emits validated code)	Medium (AWS maintains skill + SDK)
Hybrid	Medium (agent uses skill for common tasks, SDK for edge cases)	High	Medium

The skill trades maintenance burden for agent simplicity. AWS must keep the skill in sync with SDK changes, but agents get reliable code generation without learning SageMaker internals.

Technical Verdict

Use AWS Agent Toolkit skills when:

• You are building coding agents that need deep platform expertise (SageMaker, ECS, Lambda optimization)
• You want agents to generate executable code, not just API calls
• You can tolerate the dependency on AWS-maintained skill packages
• Your agents run in environments where you control SDK versions

Avoid when:

• You need cross-cloud portability (skills are AWS-specific)
• Your agents must work offline or in air-gapped environments
• You require full control over generated code patterns
• Your use case involves edge cases the skill does not cover

The skill pattern works best for high-complexity, high-value tasks where the cost of teaching agents platform internals exceeds the cost of maintaining a skill package. For simple API calls, direct SDK access is simpler.

Source Links

• AWS Machine Learning Blog: New agent skill for SageMaker inference