mech.app

The mech.app newsletter

Agentic AI, minus the noise.

Get practical field notes on AI agents, automation, developer tools and security delivered to your inbox.

No spam. Unsubscribe anytime.

Security

Ironic-Python-Agent Container Escape: What OpenStack's Security Flaw Reveals About Agent Isolation Boundaries

OpenStack's bare-metal provisioning agent shipped with broken container isolation. Here's what it teaches about hardening agent runtime environments.

Source: seclists.org
Ironic-Python-Agent Container Escape: What OpenStack's Security Flaw Reveals About Agent Isolation Boundaries

OpenStack’s Ironic-Python-Agent shipped a container-based plugin system in 2025.2 that ignored its own security controls. The Container HardwareManager feature let operators run arbitrary containers during bare-metal provisioning, but the implementation bypassed the allow_arbitrary_containers safety flag and failed to enforce isolation boundaries. The result: a malicious or compromised plugin could escape the container and compromise the provisioning infrastructure.

This is not a theoretical supply-chain risk. Ironic-Python-Agent runs on bare-metal nodes during deployment, cleaning, and servicing. It has elevated privileges by design because it needs to configure hardware, write disk images, and manage firmware. The security model assumed container isolation would prevent a rogue HardwareManager from escalating to the host. That assumption was wrong.

What Ironic-Python-Agent Does

Ironic is OpenStack’s bare-metal provisioning service. When you provision a physical server, Ironic boots a minimal Linux image (the ramdisk) on the target machine. That ramdisk runs Ironic-Python-Agent (IPA), which acts as the control plane’s agent on the hardware.

IPA uses a plugin architecture called HardwareManagers. Each HardwareManager exposes steps that Ironic can invoke during:

  • Cleaning: wiping disks, resetting BIOS settings
  • Deployment: writing OS images, configuring RAID
  • Servicing: firmware updates, diagnostics

The Container HardwareManager introduced in 2025.2 added the ability to download and execute containers as Ironic steps. This lets operators package custom provisioning logic (vendor tools, proprietary firmware updaters) as container images instead of baking them into the ramdisk.

The Security Model That Failed

The intended security boundary:

  1. IPA runs on the bare-metal node with elevated privileges (it needs to write raw disks, manage hardware).
  2. Container HardwareManager downloads a container image and runs it using podman or docker.
  3. The container runs with restricted capabilities, isolated from the host.
  4. A configuration flag (allow_arbitrary_containers) controls whether operators can specify any container image or only pre-approved ones.

The actual implementation:

  • Ignored the allow_arbitrary_containers flag entirely.
  • Did not enforce seccomp profiles, AppArmor policies, or namespace restrictions.
  • Mounted host paths into containers without validation.
  • Allowed containers to run with --privileged or equivalent access.

A malicious HardwareManager plugin could:

  • Escape the container using standard breakout techniques (exposed Docker socket, writable cgroup mounts, kernel exploits).
  • Modify the host ramdisk to persist across reboots.
  • Exfiltrate credentials or cryptographic material used by IPA to authenticate to the Ironic API.
  • Compromise the provisioning network and pivot to other nodes.

Why This Matters for Agent Infrastructure

Most agentic systems assume the agent runtime is trusted. You secure the orchestration layer, the tool call boundaries, the data plane, but the agent process itself runs in a privileged context because it needs to interact with external systems.

Ironic-Python-Agent is an extreme case: it runs on bare metal with root access. But the pattern is common:

  • CI/CD agents (Jenkins, GitLab Runner) execute arbitrary code from repositories.
  • Infrastructure agents (Ansible, Puppet) apply configuration with elevated privileges.
  • Observability agents (Datadog, New Relic) scrape metrics and logs with filesystem access.
  • AI agents with tool-calling capabilities execute shell commands, API calls, or database queries.

When you add a plugin system to an agent, you create a new attack surface. The plugin runs inside the agent’s security context. If the agent has elevated privileges, the plugin inherits them.

Isolation Primitives That Actually Work

Container isolation is not a security boundary unless you enforce it. Here’s what you need:

PrimitivePurposeTrade-off
Seccomp profilesBlock dangerous syscalls (mount, reboot, kernel modules)Breaks plugins that need hardware access
AppArmor/SELinuxRestrict file and network access by policyRequires per-plugin policy tuning
User namespacesMap container root to unprivileged host UIDIncompatible with some volume mounts
Read-only root filesystemPrevent container from modifying itselfPlugin must write to explicit volumes
No host network/PID/IPCIsolate from host processes and socketsBreaks plugins that need host visibility
Capability droppingRemove CAP_SYS_ADMIN, CAP_NET_ADMIN, etc.Plugin may need specific capabilities

The Ironic fix applies these controls:

# Ironic 2026.2+ enforces container restrictions
[container]
allow_arbitrary_containers = false
approved_images = registry.example.com/approved/*

# Podman run flags enforced by IPA
--security-opt no-new-privileges
--security-opt seccomp=ironic-hwm.json
--cap-drop ALL
--cap-add NET_BIND_SERVICE  # only if needed
--read-only
--tmpfs /tmp:rw,noexec,nosuid
--network none  # unless plugin needs network

The seccomp profile blocks mount, reboot, and kernel module syscalls. The container runs with a read-only root filesystem and no capabilities except those explicitly granted. Network access is disabled by default.

Auditing Third-Party Plugins

You cannot trust a plugin just because it ships as a container. The container image is the distribution format, not the security boundary. Before you load a third-party HardwareManager:

Static analysis:

  • Inspect the Dockerfile. Does it run as root? Does it install debugging tools (gdb, strace) that could be weaponized?
  • Check the entrypoint script. Does it download additional code at runtime? Does it modify system files?
  • Scan for known vulnerabilities in base images and dependencies.

Runtime constraints:

  • Pin the image digest, not the tag. registry.example.com/plugin@sha256:abc123 prevents tag hijacking.
  • Use a private registry with image signing (Notary, Cosign). Reject unsigned images.
  • Run the plugin in a test environment first. Monitor syscalls, file access, and network connections.

Least privilege:

  • Grant only the capabilities the plugin needs. If it writes firmware, it needs /dev/mem access. If it configures network interfaces, it needs CAP_NET_ADMIN. Nothing else.
  • Use a dedicated service account with limited Ironic API permissions. The plugin should not be able to provision arbitrary nodes or modify global settings.

Observability:

  • Log all container executions: image digest, command, exit code, duration.
  • Alert on unexpected syscalls (seccomp violations), file modifications outside approved paths, or network connections.
  • Collect container logs and forward them to a centralized system. Do not rely on logs stored in the container filesystem.

The Backwards Compatibility Trap

Ironic did not backport the security fixes to 2025.2 because they break existing deployments. Operators who built custom HardwareManagers that rely on privileged access or host mounts would see their plugins fail after the upgrade.

This is the cost of shipping insecure defaults. Once users depend on the broken behavior, you cannot fix it without breaking their workflows. The Ironic team chose to:

  • Ship the fix in 2026.2 as a breaking change.
  • Provide a configuration option to disable the Container HardwareManager entirely in older releases.
  • Document the migration path for operators who need the feature.

If you are building an agent plugin system, enforce security boundaries from day one. Do not ship a permissive mode and promise to lock it down later. Users will depend on the permissive mode, and you will be stuck with it.

Technical Verdict

Use container-based agent plugins when:

  • You need to distribute vendor-specific logic without modifying the agent codebase.
  • You can enforce strict isolation (seccomp, capabilities, read-only filesystem).
  • You control the plugin registry and can audit images before deployment.
  • The agent does not run with elevated privileges, or you can drop privileges before loading the plugin.

Avoid container-based agent plugins when:

  • The agent already has root or equivalent access, and the plugin inherits it.
  • You cannot enforce isolation without breaking legitimate plugin functionality.
  • You rely on third-party plugins from untrusted sources.
  • Your threat model includes supply-chain attacks on the plugin distribution mechanism.

For Ironic specifically: if you use custom ramdisks with podman or docker installed, either upgrade to 2026.2 or disable the Container HardwareManager by adding its steps to disallow_service_steps, disallow_clean_steps, and disallow_deploy_steps. Do not assume the container provides a security boundary. It does not.