Scientific workflow management systems (WMSs) automate execution but rely on hand-tuned, rule-based orchestration. Introducing LLM agents into these pipelines promises autonomy, but raises hard questions: which decisions are safe to delegate, how to contain failures, and whether agentic reasoning actually improves outcomes. Avatar, a new actor-based architecture from Argonne and the University of Chicago, addresses these questions by treating orchestration policies as pluggable adapters. The same core can run deterministic rules or LLM-backed reasoning, letting teams test agentic control without rewriting their workflow systems.
The research demonstrates a 55% reduction in compute waste and 40% cut in GPU-busy time when LLM agents replace fixed rules in three production workloads. The key insight is not that agents are always better, but that Avatar identifies where they add value and where they do not.
The Orchestration Boundary Problem
Traditional workflow systems separate execution from orchestration. The executor runs tasks. The orchestrator decides what runs next, on which resources, and when to retry or abort. These decisions are typically encoded as static rules: if task A fails twice, scale up memory and retry; if GPU utilization drops below 50%, queue the next batch.
This works until workload characteristics shift. A rule tuned for one dataset size breaks on another. A retry policy optimized for transient network errors wastes resources on persistent configuration bugs. Engineers spend time tuning rules that only work for narrow conditions.
LLM agents can reason about context, but introducing them into production pipelines creates new risks:
- Non-determinism: LLM outputs vary across runs, breaking reproducibility guarantees required for scientific workflows.
- Latency: Agent reasoning adds seconds or minutes to orchestration decisions that previously took milliseconds.
- Failure modes: Agents can hallucinate invalid actions, misinterpret provenance data, or make decisions that violate resource constraints.
Avatar addresses these risks by treating orchestration policies as swappable components. The same workflow can run with rule-based policies in production and LLM-backed policies in staging, using identical execution infrastructure.
Avatar Architecture
Avatar decomposes orchestration into three actors:
- Orchestrator: Decides which tasks to schedule, on which resources, and in what order.
- Executor: Runs tasks and reports outcomes (success, failure, resource usage).
- Provenance Monitor: Tracks execution history, resource consumption, and task dependencies.
Each actor exposes a validated action catalog. The orchestrator can emit schedule_task, retry_task, or abort_workflow. The executor can emit task_started, task_completed, or task_failed. The provenance monitor can emit log_event or query_history.
The decision policy for each actor is pluggable. A rule-based policy uses if-then logic. An LLM-backed policy calls a language model with the current state, action catalog, and provenance history, then validates the response against the catalog before executing.
class OrchestratorActor:
def __init__(self, policy: Policy, action_catalog: ActionCatalog):
self.policy = policy
self.catalog = action_catalog
def decide(self, state: WorkflowState, provenance: ProvenanceLog):
# Policy can be rule-based or LLM-backed
proposed_action = self.policy.choose_action(state, provenance)
# Validate against catalog before executing
if self.catalog.is_valid(proposed_action):
return proposed_action
else:
# Fallback to safe default or raise error
return self.catalog.get_default_action(state)This design isolates agentic reasoning from execution. If an LLM hallucinates an invalid action, the catalog rejects it. If an LLM takes too long, the orchestrator can fall back to a rule-based policy. The executor never sees the difference.
Where Agents Add Value
Avatar evaluates three workloads:
- Molecular dynamics simulation: 100-task pipeline with variable task durations and GPU requirements.
- Climate model ensemble: 500-task workflow with data dependencies and checkpoint/restart logic.
- Genomics pipeline: 1,000-task DAG with heterogeneous resource needs (CPU-bound alignment, GPU-bound variant calling).
The LLM-backed orchestrator outperforms rule-based policies in two scenarios:
Dynamic resource allocation: The molecular dynamics workload has tasks that vary in GPU memory requirements (2GB to 16GB). Rule-based policies allocate maximum memory to every task, wasting resources. The LLM agent analyzes task input size and previous runs, then allocates just enough memory. This cuts GPU-busy time by 40% because more tasks fit on each GPU.
Adaptive retry logic: The climate model workload has transient filesystem errors (NFS timeouts) and persistent errors (corrupted checkpoint files). Rule-based policies retry every failure three times, wasting hours on persistent errors. The LLM agent examines error messages and provenance logs, then skips retries for corrupted checkpoints and immediately retries NFS timeouts. This reduces compute waste by 55%.
The genomics pipeline shows no improvement. Task dependencies are fixed, resource requirements are predictable, and the rule-based policy already handles failures correctly. Adding LLM reasoning adds latency without benefit.
Failure Containment
Avatar contains agent failures through three mechanisms:
Action catalog validation: Every action must match a schema. If an LLM proposes schedule_task(task_id="nonexistent", resources={"gpu": -1}), the catalog rejects it. The orchestrator logs the error and falls back to a safe default (skip the task or abort the workflow).
Timeout and fallback: LLM calls have a 30-second timeout. If the agent does not respond, the orchestrator switches to rule-based policy for that decision. The workflow continues without blocking.
Provenance auditing: Every decision (rule-based or LLM-backed) is logged with input state, reasoning trace, and outcome. If an LLM decision causes a failure, engineers can replay the decision with a different policy or model.
The paper reports zero workflow aborts due to agent failures across 1,600 total tasks. Three LLM calls timed out (0.2% rate), triggering fallback to rule-based policies. Two LLM calls proposed invalid actions (0.1% rate), rejected by catalog validation.
Implementation on Academy Framework
Avatar runs on Academy, a Python-based workflow framework. The core change is replacing Academy’s fixed orchestration loop with the actor model:
# Before: Fixed orchestration loop
while tasks_remaining:
task = select_next_task(ready_tasks)
executor.submit(task)
wait_for_completion()
# After: Actor-based orchestration
orchestrator = OrchestratorActor(policy=llm_policy, catalog=action_catalog)
executor = ExecutorActor()
monitor = ProvenanceMonitor()
while not orchestrator.is_done():
state = monitor.get_current_state()
action = orchestrator.decide(state, monitor.get_log())
executor.execute(action)
monitor.log(action, executor.get_result())The LLM policy uses GPT-4 with a 4,000-token context window. The prompt includes:
- Current workflow state (tasks completed, tasks ready, tasks blocked)
- Resource availability (GPUs free, memory available)
- Recent provenance (last 10 task outcomes, error messages)
- Action catalog (valid actions and their schemas)
The model returns a JSON action. If parsing fails, the catalog rejects it. If the action is valid, the executor runs it.
Trade-offs and Risks
| Dimension | Rule-Based Policy | LLM-Backed Policy |
|---|---|---|
| Latency | <1ms per decision | 2-5s per decision (LLM call) |
| Determinism | Fully reproducible | Non-deterministic (model updates, sampling) |
| Adaptability | Requires manual tuning | Adapts to new conditions |
| Failure modes | Predictable (logic bugs) | Unpredictable (hallucinations, timeouts) |
| Cost | Zero marginal cost | $0.01-0.10 per decision (API calls) |
| Auditability | Rule trace is explicit | Requires logging LLM reasoning |
The 2-5 second latency is acceptable for scientific workflows where tasks run for minutes or hours. It would break interactive systems or sub-second orchestration loops.
Non-determinism is manageable because Avatar logs every decision. If a workflow produces different results across runs, engineers can replay decisions with the same LLM inputs and compare outputs. This is harder than debugging rule-based policies but better than no auditability.
Cost is low for the evaluated workloads (1,600 tasks, 1,600 LLM calls, ~$16 total). It scales linearly with task count, so a 100,000-task workflow would cost $1,000 in LLM calls. This is acceptable for workflows that run on $10,000+ of compute but prohibitive for lightweight pipelines.
When to Use Avatar
Use Avatar when:
- Workflow orchestration rules are brittle and require frequent tuning.
- Workload characteristics vary unpredictably (task durations, resource needs, failure patterns).
- You can tolerate 2-5 second orchestration latency.
- You need auditability and can log LLM reasoning traces.
- Compute costs dwarf LLM API costs.
Avoid Avatar when:
- Orchestration decisions must be deterministic (regulatory compliance, bit-exact reproducibility).
- Latency requirements are sub-second.
- Workload characteristics are stable and rule-based policies work well.
- LLM API costs exceed compute savings.
- You cannot validate LLM outputs against a schema.
The actor-based architecture is useful even without LLM agents. Treating orchestration policies as pluggable components lets teams test new policies (rule-based or agentic) without rewriting workflow systems. The action catalog provides a safety boundary that works for any policy type.
Technical Verdict
Avatar solves a real problem: where to introduce agentic reasoning in deterministic workflow systems without destabilizing production pipelines. The actor model with validated action catalogs provides a clean boundary between policy (rule-based or LLM-backed) and execution. The results are credible: 55% compute waste reduction and 40% GPU-busy time reduction are significant, and the failure containment mechanisms (catalog validation, timeout, fallback) are sound.
The approach works best for workflows with variable resource needs and unpredictable failure modes. It does not help workflows where rule-based policies already perform well. The 2-5 second orchestration latency is acceptable for scientific pipelines but breaks interactive or real-time systems.
The architecture is portable. Any workflow system can adopt the actor model and action catalog pattern without depending on Avatar’s specific implementation. The key insight is treating orchestration policies as adapters, not core logic.
Use Avatar when you are tuning orchestration rules weekly and workload characteristics shift faster than you can update rules. Skip it when your workflows are stable and deterministic execution is non-negotiable.