Tracecat: Security Alert Orchestration Plumbing for SOAR Workflows

Security alert automation sits at the intersection of workflow orchestration, credential management, and multi-tool integration. When an alert fires, a security analyst must contact victims, query logs across three SIEMs, check vulnerability databases, update tickets, and notify stakeholders. Doing this manually for 100 alerts per day is unsustainable. Doing it with brittle scripts creates new failure modes.

Tracecat is an open-source SOAR (Security Orchestration, Automation, and Response) platform that exposes the plumbing required to chain security tools into reliable investigation workflows. It replaces commercial platforms like Splunk SOAR and Palo Alto Cortex XSOAR with a self-hosted stack built on temporal task execution, declarative workflow definitions, and pluggable integrations.

Architecture: Temporal Tasks and Workflow DAGs

Tracecat runs workflows as directed acyclic graphs (DAGs) where each node is a task executed by a Temporal worker. This is not a custom scheduler. The platform delegates task queuing, retries, and state persistence to Temporal, which handles distributed execution and failure recovery.

Core components:

• Workflow engine: Parses YAML workflow definitions into Temporal workflow executions
• Action registry: Catalog of pre-built integrations (SIEM queries, ticket creation, email sending)
• Secret store: Encrypted credential vault for API keys and service accounts
• Event ingestion: Webhook endpoints that accept alerts from detection tools
• UI builder: Drag-and-drop interface for constructing investigation playbooks

When an alert arrives via webhook, the ingestion service validates the payload, extracts metadata (alert type, severity, affected assets), and starts a Temporal workflow. Each workflow step is a task that calls an external tool, transforms data, or makes a routing decision. Temporal guarantees exactly-once execution and persists workflow state to a database.

Alert Routing and Investigation Chains

A typical investigation workflow looks like this:

1. Alert arrives from a SIEM or EDR tool via webhook
2. Enrichment step queries threat intelligence APIs for IOC context
3. Log aggregation pulls relevant events from multiple log sources
4. Decision node routes high-severity alerts to human analysts, auto-closes false positives
5. Notification step sends Slack messages or creates Jira tickets
6. Victim contact emails affected users with remediation instructions

Each step is a Temporal activity. If the threat intelligence API times out, Temporal retries with exponential backoff. If the workflow crashes mid-execution, it resumes from the last completed activity when the worker restarts.

Workflow definition example:

name: phishing_investigation
trigger:
  type: webhook
  path: /alerts/phishing
steps:
  - id: enrich_ioc
    action: virustotal.lookup_url
    inputs:
      url: ${{ trigger.payload.suspicious_url }}
    timeout: 30s
  - id: query_logs
    action: splunk.search
    inputs:
      query: "index=email url=${{ trigger.payload.suspicious_url }}"
      time_range: "-24h"
    timeout: 120s
  - id: create_ticket
    action: jira.create_issue
    inputs:
      project: SEC
      summary: "Phishing alert: ${{ trigger.payload.subject }}"
      description: ${{ steps.enrich_ioc.output.report }}
  - id: notify_team
    action: slack.send_message
    inputs:
      channel: "#security-alerts"
      text: "New phishing case: ${{ steps.create_ticket.output.issue_key }}"

The workflow engine interpolates variables from trigger payloads and previous step outputs. If enrich_ioc fails, the workflow does not proceed to query_logs. If create_ticket succeeds but notify_team fails, Temporal retries only the notification step.

Credential Management and Tool Authentication

Security workflows touch dozens of APIs, each requiring different authentication methods (API keys, OAuth tokens, service account credentials). Tracecat stores secrets in an encrypted vault backed by PostgreSQL or HashiCorp Vault.

Secret injection flow:

1. Admin creates a secret in the UI (e.g., VIRUSTOTAL_API_KEY)
2. Workflow references the secret by name: api_key: ${{ secrets.VIRUSTOTAL_API_KEY }}
3. At runtime, the workflow engine fetches the decrypted value and injects it into the activity context
4. The activity function receives the credential as an environment variable, never logs it

Secrets are scoped to workspaces. A workflow in the incident_response workspace cannot access secrets from the threat_hunting workspace. This prevents credential leakage across team boundaries.

State Persistence for Long-Running Investigations

Some investigations span hours or days. A ransomware incident might require:

• Initial containment (minutes)
• Forensic analysis (hours)
• Victim notification (days, waiting for legal approval)

Temporal persists workflow state to a database after each activity completes. If the Tracecat server restarts during a multi-day investigation, the workflow resumes from the last checkpoint. The analyst does not lose progress.

State checkpointing:

• After each activity, Temporal writes the output to the workflow history
• The workflow can sleep for arbitrary durations using time.sleep activities
• External systems can query workflow status via API: “Is the victim notification step complete?”

This is different from stateless automation tools like Zapier, where a workflow either completes in one execution or fails. SOAR workflows require durable state because investigations do not fit into a single request-response cycle.

Failure Modes and Error Handling

Security automation failures have real consequences. A missed alert could mean undetected data exfiltration. A misconfigured workflow could spam victims with duplicate emails.

Common failure scenarios:

Failure Type	Impact	Mitigation
SIEM API timeout	Log query incomplete, investigation stalls	Temporal retry with backoff, fallback to secondary SIEM
Credential expiration	All workflows fail authentication	Secret rotation alerts, pre-expiry warnings
Webhook flood	Ingestion queue saturates, alerts dropped	Rate limiting, priority queues for high-severity alerts
Workflow logic error	Infinite loop or resource exhaustion	Execution time limits, circuit breakers
Database unavailable	Workflow state lost, cannot resume	Temporal’s built-in persistence layer handles DB failover

Tracecat exposes observability hooks for each failure mode. Workflows emit structured logs to stdout, which can be shipped to a centralized logging system. Temporal’s UI shows workflow execution history, including retry attempts and error stack traces.

Deployment Shape and Scaling Boundaries

A minimal Tracecat deployment requires:

• Temporal cluster (3 nodes for HA)
• PostgreSQL database (workflow state, secrets, audit logs)
• Tracecat API server (workflow management, webhook ingestion)
• Temporal workers (execute activities, scale horizontally)

For a team handling 1,000 alerts per day, a single worker node can process workflows sequentially. For 10,000 alerts per day, horizontal scaling adds more workers. Temporal distributes tasks across the worker pool using a task queue.

Scaling bottlenecks:

• Database writes: Each activity completion writes to the workflow history. High-throughput workflows need connection pooling and write replicas.
• External API rate limits: If 100 workflows query VirusTotal simultaneously, the API quota exhausts. Workers need rate-limiting middleware.
• Secret decryption: Fetching secrets from Vault on every activity invocation adds latency. Workers cache decrypted secrets with TTL-based invalidation.

Observability and Audit Trails

Security teams need to answer questions like:

• “Which analyst approved the containment action?”
• “Did the workflow notify all affected users?”
• “Why did the investigation take 6 hours instead of 30 minutes?”

Tracecat logs every workflow execution, activity input/output, and decision point. The audit trail is immutable and stored separately from operational logs.

Audit log schema:

{
  "workflow_id": "phishing-inv-2024-03-25-001",
  "step": "enrich_ioc",
  "timestamp": "2024-03-25T18:12:34Z",
  "actor": "analyst@example.com",
  "action": "virustotal.lookup_url",
  "inputs": {"url": "https://evil.example"},
  "outputs": {"malicious": true, "detections": 42},
  "duration_ms": 1234
}

This log can be queried to reconstruct the investigation timeline. If a workflow makes a mistake (e.g., emails the wrong user), the audit trail shows which step produced the incorrect output.

Integration Patterns and Tool Chaining

SOAR platforms are only useful if they integrate with existing security tools. Tracecat ships with 50+ pre-built integrations (SIEMs, ticketing systems, threat intelligence APIs, email gateways).

Integration types:

• Polling integrations: Periodically check for new alerts (e.g., query Jira for unassigned tickets)
• Webhook integrations: Receive push notifications from external tools
• Action integrations: Execute commands in external systems (e.g., block an IP in a firewall)

Custom integrations are Python functions that implement the Activity interface. A VirusTotal integration looks like this:

from temporalio import activity
import requests

@activity.defn
async def lookup_url(url: str, api_key: str) -> dict:
    response = requests.get(
        f"https://www.virustotal.com/api/v3/urls/{url}",
        headers={"x-apikey": api_key},
        timeout=30
    )
    response.raise_for_status()
    return response.json()

The workflow engine discovers activities via a registry. Adding a new integration means writing a Python function and registering it in the action catalog.

When to Use Tracecat

Good fit:

• Security teams that handle repetitive alert triage (phishing, malware, vulnerability scanning)
• Organizations that need audit trails for compliance (SOC 2, ISO 27001)
• Teams that want to self-host SOAR without vendor lock-in
• Workflows that chain 5+ tools (SIEM, ticketing, threat intel, email, Slack)

Poor fit:

• Teams with fewer than 50 alerts per week (manual triage is faster)
• Workflows that require sub-second latency (Temporal adds overhead)
• Organizations without Kubernetes or container orchestration experience
• Use cases that need real-time streaming (Tracecat is batch-oriented)

Technical Verdict

Tracecat exposes the real plumbing of security orchestration: durable task execution, secret injection, error recovery, and audit logging. It trades the simplicity of a hosted platform (Tines, Splunk SOAR) for full control over the execution environment. The Temporal foundation handles the hard parts of distributed workflows (retries, state persistence, worker scaling), but you still need to operate a Temporal cluster and manage database backups.

Use it when you need transparent, auditable automation for high-stakes security workflows. Avoid it if you want a no-ops SaaS experience or need real-time event processing. The open-source model means you can inspect every line of orchestration logic, but you own the operational burden.

Source Links

• Tracecat GitHub Repository
• Hacker News Discussion (264 points, 65 comments)