See also — Parent: docs/README.md, orchestration.md (Layer 5) · Autonomy mapping: compliance.md —
dagengine covers A0–A1;delegation_loopengine covers A2–A4 · Delegation-loop concepts: manager-intelligence.md (PLAN/DELEGATE/COMPLETE decisions), critique.md (per-worker repair), runtime.md (budget envelope, completion gate chain, R35 repair fixpoint) · DAG-engine concepts: memory.md (share_output), validation.md (graph/cycle rules) · Observability: observability.md · Spec: spec/versions/1.0/layers/05-orchestration.md
AWP ships two orchestration engines because predictable pipelines and open-ended exploration have fundamentally different control structures, and forcing both into a single abstraction makes both worse.
- The DAG engine trusts the workflow author to know the steps. Edges are explicit, the graph is static, and the runtime just walks it in topological order. Cost is
N agents * cost_per_agent. Auditing is reading a YAML file. - The Delegation Loop engine trusts a manager LLM to discover the steps from the task itself, but bounds that trust with a deterministic safety envelope (budgets, forbidden tools, sandbox limits, reservation-based budget arithmetic). Cost is bounded by
budget.max_*. Auditing is reading a per-iteration audit trail on disk.
Both engines share everything else: the agent contract (R17 confidence output), the tool registry, the memory tiers, the validators, the observability layer. Choose by the structure of the work, not by autonomy level for its own sake — though as a heuristic, A0-A1 is DAG territory and A2-A4 is delegation-loop territory.
The two engines also compose: a DAG node can be a delegation loop, giving you a predictable outer pipeline with a flexible inner step. See Pattern 4.
Related reading: orchestration.md (DAG field reference + delegation-loop config), manifest.md (envelope declaration), manager-intelligence.md (auto-promotion logic), critique.md (reflective repair loop), runtime-tool-generation.md (B1-B6 tool pipeline).
Each engine answers the same question — "In what order and under what conditions do agents run?" — but with fundamentally different philosophies.
Status: Available since AWP 1.0.0 Best for: Predictable pipelines with known steps
You define a static graph of agents and their dependencies in workflow.awp.yaml.
The runtime topologically sorts the graph into execution levels and runs agents
in order, respecting dependencies.
orchestration:
engine: dag
graph:
- id: planner
agent: planner
depends_on: []
- id: researcher
agent: researcher
depends_on: [planner]
- id: writer
agent: writer
depends_on: [researcher]Agents at the same level with no mutual dependencies run in parallel.
| Feature | Support |
|---|---|
| Sequential execution | Yes |
| Parallel execution | Yes (same-level, independent agents) |
| Conditional execution | Yes (when expressions) |
| State sharing | Yes (full, selective, isolated) |
| Retry with backoff | Yes |
| Fan-out | Yes (via FanOutConfig) |
| Loops | Yes (per-agent LoopConfig) |
| Tools | Yes (full MCP tool registry) |
| Code Mode | Yes |
| Dynamic Tool Creation | Yes |
- The number of steps is known before execution
- Each agent's role is fixed and well-defined
- Data flows in one direction (pipeline pattern)
- You need maximum predictability and auditability
- Budget planning is straightforward (N agents × cost per agent)
orchestration:
engine: dag
execution:
mode: parallel # sequential | parallel | conditional
timeout:
per_agent: 120 # seconds
total: 300
max_parallel_agents: 4
error_handling:
default: continue # continue | stop | retry
max_retries: 1
retry_delay: 2.0
graph:
- id: agent_name
agent: agent_dir_name
enabled: true
depends_on: [other_agent]
share_output: [field1, field2]
on_failure: continue # continue | skip | abort
when: "state.other_agent.score > 0.5"
timeout: 60
retry: 2Status: Available since AWP 1.1.0 Best for: Open-ended tasks where the steps emerge during execution
A manager agent receives the task and dynamically decides what to do:
- DELEGATE — Generate instructions, skills, and tool configs for worker agents
- COMPLETE — The task is done, return the final result
- FAIL — The task cannot be completed
Workers are ephemeral — they don't exist as static agent.awp.yaml files.
The manager creates them at runtime by generating a Delegation Envelope
containing everything the worker needs: instructions, domain knowledge (skills),
allowed tools, and output contract.
The manager generates this for each worker:
{
"worker_id": "market_researcher",
"instructions": "Research current market trends for electric vehicles...",
"skills": [
"## Market Analysis\n- Focus on TAM/SAM/SOM...\n- Use Porter's Five Forces..."
],
"tools_allowed": ["web.search", "file.read"],
"output_contract": {
"required_fields": ["trends", "market_size", "confidence"],
"description": "Market research findings"
},
"codemode": {
"enabled": false,
"tool_creation": false
}
}The manager dynamically generates:
- Instructions — What the worker should do
- Skills — Domain knowledge as Markdown, injected into the worker's prompt
- Tool configuration — Which MCP tools the worker can use
- Output contract — What the worker must return
Instead of a simple depth limit, the delegation loop uses a budget system that tracks resource consumption across the entire execution tree:
budget:
max_loops: 20 # Maximum manager iterations
max_total_workers: 30 # Total workers across all iterations
max_total_tokens: 1000000 # LLM token limit
max_wall_time: 600 # Wall clock seconds
max_tool_calls: 200 # Total tool invocations
max_depth: 2 # Safety limit for recursive sub-delegation (default 2)
max_concurrent_submanagers: 3 # Max submanagers running at once (default 3)
max_total_submanagers_per_run: 6 # Hard cap on submanagers per run (default 6)The budget enforces a hard stop when any resource limit is hit. This prevents runaway loops and guarantees cost predictability.
When workers sub-delegate (recursive delegation), they must allocate portions
of their budget to sub-workers. The invariant sum(children) + self <= allocation
is enforced deterministically, not by the LLM.
The manager controls worker instructions, skills, and tools — but it cannot override security-critical parameters:
worker_policy:
enforced: # Manager CANNOT change these
sandbox:
type: subprocess
max_memory_mb: 512
max_cpu_seconds: 30
network: false
codemode:
max_tools_per_worker: 10
rate_limiting:
max_llm_calls_per_minute: 30
forbidden_tools:
- "shell.execute"
manager_controlled: # Manager CAN set these
- instructions
- skills
- tools_allowed
- output_contract
- codemode.enabled
- codemode.tool_creation
- temperature # Per-worker LLM temperatureThe model selection is also outside the manager's control — it's set by the user
via --manager-model and --worker-model CLI flags. However, the manager can
set temperature per worker in the delegation envelope to control creativity
(e.g., low for analysis, high for brainstorming). If omitted, defaults to 0.2.
Every worker result passes through two validation stages:
Tier 1: Deterministic (always runs, free)
- Is the result a valid JSON object?
- Does it contain the required
confidencefield? - Is confidence in range [0.0, 1.0]?
- Is the budget still within limits?
Tier 2: LLM Semantic (conditional, costs tokens)
- Does the result actually address the task?
- Is the confidence score realistic?
- Does it contradict previous findings?
The LLM validator runs on the worker model (not the expensive manager model) and can be skipped when confidence is very high or budget is low.
The loop automatically detects when it's making no progress:
termination:
enabled: true
window: 3 # Compare last 3 iterations
min_confidence_delta: 0.05 # Must improve by at least 0.05
action: warn_then_stop # warn once, stop on second stallIf confidence doesn't improve over window iterations, the system:
- First occurrence: warns the manager ("confidence is stagnating")
- Second occurrence: terminates the loop with partial results
Worker results can be large (DataFrames, analysis outputs, etc.). The context budget controls how much data is inlined into worker prompts vs spilled to files:
context_budget:
total_chars: 64000 # Total character budget for all context (~16K tokens)
min_per_entry: 4000 # Floor per entry even with many workers
preview_chars: 2000 # Preview length for spilled resultsAuto-detect (default): The total_chars budget is divided equally among the
number of context entries. Each entry gets at least min_per_entry characters.
Spillover behavior:
- Results under budget: inlined fully as JSON in the worker prompt
- Results over budget: written to
workspace/context/<worker_id>.json, with a truncated preview and file path shown in the prompt - Workers can read the full spilled file via
code.execute:json.load(open(_workspace_dir + "/context/worker_1.json"))
Input Registry: Workers automatically receive a listing of all available
data files in workspace/inputs/ and workspace/context/, including:
- File name, size, and access path
- Schema preview for CSV files (column names, first 3 rows, row count)
- Structure preview for JSON files (top-level keys, array sizes)
- Read hints for binary formats (Parquet, Feather, etc.)
This is configured at the orchestration level (context_budget field) and
applies to both DAG and delegation loop engines.
To prevent context window overflow, the loop maintains a rolling summary:
- Last N iterations: Full results (configurable via
full_results_window) - Older iterations: Summarized as key findings + confidence
- On disk:
ROLLING_SUMMARY.md(human-readable) +rolling_summary.json(machine-readable)
The manager receives the rolling summary in every iteration, keeping its context size constant regardless of how many iterations have run.
Every run produces a complete audit trail on disk:
workspace/runs/{run_id}/
├── RUN_SUMMARY.md # Human-readable overview
├── run_manifest.json # Machine-readable config
├── run_completion.json # Final status and budget
│
├── iterations/
│ ├── 001/
│ │ ├── ITERATION_SUMMARY.md # What happened
│ │ ├── manager_decision.json # Manager's raw output
│ │ ├── budget_snapshot.json # Budget after this iteration
│ │ ├── validation.json # Validation results
│ │ └── delegations/
│ │ ├── worker_a/
│ │ │ ├── envelope.json # What the worker received
│ │ │ ├── result.json # What the worker returned
│ │ │ ├── RESULT.md # Human-readable result
│ │ │ └── generated_skills/
│ │ │ └── skill_0.md
│ │ └── worker_b/
│ │ └── ...
│ └── 002/
│ └── ...
│
├── history/
│ ├── ROLLING_SUMMARY.md # Current summary
│ └── rolling_summary.json # Machine-readable history
│
└── artifacts/
├── skills/ # All generated skills
└── tools/ # All generated tools
Both JSON (for machines/agents) and Markdown (for humans/debugging) are generated
by default. Use logging.format: json or logging.format: md for single-format.
The user controls which models are used:
# Manager gets the strong model, workers get the fast model
awp run my-workflow/ \
--task "Analyze market trends" \
--manager-model openrouter/anthropic/claude-opus-4 \
--worker-model openrouter/anthropic/claude-sonnet-4Resolution order:
--manager-model → workflow YAML models.manager → LLM_MODEL env → Error
--worker-model → workflow YAML models.worker → manager model → Error
The manager cannot override the worker model. This prevents a hallucinating manager from upgrading workers to expensive models.
Workers can act as managers and spawn their own sub-workers:
Manager (depth 0)
├── Worker A (depth 1) → becomes sub-manager
│ ├── Sub-Worker A1 (depth 2)
│ └── Sub-Worker A2 (depth 2)
├── Worker B (depth 1)
└── Worker C (depth 1)
This is controlled by the budget system, not just depth limits. A worker that sub-delegates must allocate portions of its budget to sub-workers.
orchestration:
engine: delegation_loop
delegation_loop:
manager: agents/manager # Agent directory for the manager
models:
manager: null # CLI → YAML → LLM_MODEL
worker: null # CLI → YAML → manager model
worker_policy:
enforced:
sandbox:
type: subprocess
max_memory_mb: 512
max_cpu_seconds: 30
network: false
codemode:
max_tools_per_worker: 10
rate_limiting:
max_llm_calls_per_minute: 30
forbidden_tools:
- "shell.execute"
manager_controlled:
- instructions
- skills
- tools_allowed
- output_contract
- codemode.enabled
- codemode.tool_creation
- temperature
budget:
max_loops: 20
max_total_workers: 30
max_total_tokens: 1000000
max_wall_time: 600
max_tool_calls: 200
context_budget:
total_chars: 64000 # auto-divided among workers
min_per_entry: 4000 # floor per entry
preview_chars: 2000 # preview for spilled results
max_depth: 5
termination:
enabled: true
window: 3
min_confidence_delta: 0.05
action: warn_then_stop
validation:
deterministic:
always: true
checks: [schema, required_fields, confidence, budget]
llm:
enabled: true
skip_when_confidence_above: 0.95
skip_when_budget_remaining_below: 0.1
history:
rolling_summary: true
full_results_window: 3
persist_to_disk: true
logging:
format: dual # dual | json | md
persist_artifacts: trueWithin a delegation-loop run, every code.execute call inside a single worker
targets the same warm Python subprocess — a long-running child managed by
PersistentExecutor. Variables, imports, and helper functions defined in one
call remain live in the next; heavy imports (numpy, pandas, matplotlib) are
paid once per worker, not once per call. Implementation details that matter
for reasoning about run behavior:
- Merged stderr/stdout. The warm child merges
stderrintostdouton the pipe level, so a chatty child process cannot deadlock on a full stderr buffer. Real tracebacks are still returned inside the JSON result envelope. - Deadline-accurate reads. Reads happen on the raw file descriptor with
select()and a private byte buffer; the configured per-call timeout is honored even if the child produces no output. - Auto-restart with history replay. If the warm subprocess dies between
calls, a fresh one is spawned and the recorded history (up to 100 entries
/ 1 MB of code text) is silently replayed so the caller sees a warm
namespace again. Timeouts (status 408) are NOT recorded — replaying a
hang would deadlock the restart. If warm execution fails at the protocol
level, the call falls back to a one-shot
subprocess.run(cold-start) and the loop continues.
Across workers, the runner maintains a per-logical-worker executor
registry. A logical id is derived by stripping repair / retry suffixes
(_repair, _retry, _vN, _strict, _final, _runN, _subtask_N) to
a fixpoint, case-insensitively. The first worker of a run claims the shared
executor; brand-new logical ids spawn their own executors. When a repair
variant of the same worker is dispatched (compose_worker_repair,
plot_runner_v2, etc.), the runtime re-enters the prior worker's
executor — the namespace, imports, and partial state from the first
attempt are still live — and prepends a ## REPAIR MODE (Continuing Prior Work) block to the worker system prompt so the LLM treats the attempt as
an incremental fix rather than a cold start.
The manager can opt out and force a clean slate via the delegation-envelope
field fresh_worker: true. This field is part of the envelope schema, not
a user-facing config flag. All per-worker executors are cleaned up in the
run finalizer; cleanup errors are logged but never propagate.
Authoritative code:
packages/awp-runtime/src/awp/runtime/persistent_executor.py,
packages/awp-runtime/src/awp/runtime/delegation_loop_runner.py
(logical_worker_id, _get_executor_for_worker, REPAIR MODE marker).
Managers and workers rarely call tool.create voluntarily, even when they
repeatedly ask code.execute to run the same shape of code with
different literal values. The delegation-loop runtime closes that gap with
a deterministic tool inducer that watches every code.execute call
and computes an AST-skeleton signature with literals normalized to
<const> and identifiers renamed to <vK> by first-occurrence index.
Structurally identical snippets share the same signature regardless of
paths, filenames, or variable names they plug in.
The inducer counts distinct worker ids per signature. When N=3
different workers have executed the same shape, the runtime:
- Lifts the varying literal slots to parameters.
- Emits a generalized Python tool and registers it through
DynamicToolFactory. - Persists it at
shared/dynamic_tools/dynamic.induced_<hash6>.jsonso later workers discover it viatool.list.
N=3 is a hardcoded constant — no new user-facing config flag was
introduced. If the varying slots cannot be cleanly parameterized (nested
expressions rather than plain constants), the inducer logs an INFO line
and skips synthesis rather than emit a broken tool. Induction never
rewrites or delays the observed code.execute call; it is pure addition.
Every emission is listed in run_completion.json.induced_tools.
Authoritative code:
packages/awp-runtime/src/awp/runtime/tool_inducer.py.
Before the manager plans the next iteration, each pending subtask is scored on a deterministic 0.0-1.0 axis:
1.0→ atomic; prefer a single flat ephemeral worker.0.0→ composite; a sub-manager with further decomposition is likely warranted.
The score is computed from description features only — atomic-keyword
count (read, compute, verify, ...), composition-keyword count (plan,
coordinate, orchestrate, synthesize, ...), declared required-outputs
count, numbered-step count in the description, and log of word count.
The exact formula and weights live in
packages/awp-runtime/src/awp/runtime/atomicity.py.
The score is advisory only. It is injected into the manager planning prompt as a hint the LLM may weigh alongside its own reasoning. The runtime MUST NOT use the score to block, override, or auto-promote a manager delegation decision — that would violate AWP's A3/A4 autonomy semantics (the manager is the authority on delegation shape). The value is surfaced for observability and tuning, not for control flow.
The cooperative budget check at the top of each manager iteration is
insufficient when a single iteration blocks indefinitely — e.g. when a
code.execute call deadlocks or an LLM client stalls. A daemon thread
named awp-walltime-watchdog is started in
DelegationLoopRunner.run() (only at depth == 0) and polls absolute
wall-time every 30 s. On breach of max_wall_time, it sends SIGTERM
to its own process, which the existing signal handler converts into a
clean status=partial exit with reason=max_wall_time and a written
run_completion.json. If the process is still alive 30 s after
SIGTERM, the watchdog escalates to SIGKILL. The watchdog is
unaffected by main-thread deadlocks — it is the structural guarantee
that max_wall_time is honored under every failure mode.
Embedded-mode guard (allow_process_kill=False): When the runner is
constructed via AgentWorkflow(allow_process_kill=False) — the path the
Studio UI server uses so the runner lives as a thread inside a long-lived
web process — the watchdog still hard-breaches on budget overrun but
swaps the SIGTERM/SIGKILL sequence for LLMClient.close_all(). All
in-flight HTTP calls raise ReadError; the runner's except/finally
blocks run as in the CLI path and produce the same
status=partial/reason=max_wall_time terminal — without killing the
host server. CLI invocations keep the original signal path.
Inside PersistentExecutor.execute() a threading.Timer is armed for
effective_timeout + 5 s before the call enters the warm-subprocess
read path. When the timer fires, a separate thread forcibly kills the
subprocess and sets _needs_replay=True + _hard_killed=True. This is
the second defense line behind α-1's select-based deadline. If the main
read thread somehow never wakes (stuck os.read, kernel-level pipe
issue), the timer still kills the child. The caller then gets a
structured cold-fallback response rather than an indefinite wait.
The warm subprocess is driven by _WORKER_SCRIPT which captures
stdout and stderr into io.StringIO buffers during exec(code, _ns). Before serialising the per-call JSON result, both buffers are
now capped at 2 MB UTF-8 bytes. Oversized output is truncated and a
...[truncated: original N bytes] marker is appended. Prior to this
cap, a print() of a multi-megabyte blob inside the worker could
overflow the 64 KB Linux pipe buffer while the parent was momentarily
busy — classic producer-consumer deadlock. The cap eliminates that
class of hang at the source, in the child, before the byte ever
touches the pipe.
The two engines can be composed. A DAG node can be a delegation loop:
orchestration:
engine: dag
graph:
- id: gather_requirements
agent: requirements_gatherer
- id: dynamic_analysis
type: delegation_loop
config:
manager: agents/analysis_manager
budget: {max_loops: 10, max_total_workers: 15}
depends_on: [gather_requirements]
- id: write_report
agent: report_writer
depends_on: [dynamic_analysis]In this pattern:
gather_requirementsruns first (DAG agent)dynamic_analysisruns as a delegation loop (multiple iterations)write_reportruns after the loop completes (DAG agent)
This gives you the predictability of a DAG for the overall flow, with the flexibility of a delegation loop for the complex middle step.
| Factor | DAG | Delegation Loop |
|---|---|---|
| Steps known upfront | Yes | No |
| Cost predictability | High (N agents) | Medium (budget-bounded) |
| Adaptability | Low | High |
| Debugging | Simple (trace the graph) | Rich (full audit trail on disk) |
| Best for | Pipelines, ETL, fixed workflows | Research, analysis, open-ended tasks |
| Agent definition | Static (agent.awp.yaml) | Manager: static, Workers: dynamic |
| LLM calls | 1 per agent | 1 per manager iteration + 1 per worker |
| Parallelism | Same-level agents | Fan-out workers per iteration |
- Start with DAG. If you can enumerate the steps, use a DAG.
- Use Delegation Loop when the number of steps depends on the task content.
- Compose both when you have a predictable outer flow with an adaptive inner step.
- Budget aggressively. Set conservative limits and increase if needed.
- Enable stall detection. It's your safety net against infinite loops.
A linear chain of agents where each builds on the previous output. Classic ETL or research-then-write flow.
orchestration:
engine: dag
graph:
- id: researcher
agent: researcher
depends_on: []
share_output: [findings]
- id: writer
agent: writer
depends_on: [researcher]
share_output: [article]Autonomy level: A1 Adaptive
Multiple agents run in parallel, then a synthesizer combines results.
orchestration:
engine: dag
graph:
- id: analyst_a
agent: analyst
depends_on: []
share_output: [analysis]
- id: analyst_b
agent: analyst
depends_on: []
share_output: [analysis]
- id: synthesizer
agent: synthesizer
depends_on: [analyst_a, analyst_b]Autonomy level: A1 Adaptive
A manager decomposes a task into subtasks and spawns workers dynamically. The number and type of workers depend on the task.
orchestration:
engine: delegation_loop
delegation_loop:
manager: agents/project_manager
budget:
max_loops: 10
max_total_workers: 20
termination:
enabled: true
window: 3
min_confidence_delta: 0.05Autonomy level: A2 Delegating
A predictable outer pipeline wraps a dynamic inner step.
orchestration:
engine: dag
graph:
- id: gather_requirements
agent: requirements_gatherer
- id: dynamic_analysis
type: delegation_loop
config:
manager: agents/analysis_manager
budget: { max_loops: 10, max_total_workers: 15 }
depends_on: [gather_requirements]
- id: write_report
agent: report_writer
depends_on: [dynamic_analysis]Autonomy level: A2 Delegating
Workers create MCP tools at runtime to handle unfamiliar APIs or data formats.
orchestration:
engine: delegation_loop
delegation_loop:
manager: agents/integration_manager
budget:
max_loops: 15
max_total_workers: 20
max_tool_calls: 300
worker_policy:
enforced:
codemode:
max_tools_per_worker: 5
forbidden_tools: ["shell.execute"]
manager_controlled:
- instructions
- skills
- tools_allowed
- codemode.enabled
- codemode.tool_creation
- temperatureAutonomy level: A3 Self-Tooling
Workers act as sub-managers, spawning their own workers. Budget flows down the hierarchy with the invariant sum(children) + self <= allocation.
Manager (depth 0, budget: 1000000 tokens)
├── Team Lead A (depth 1, budget: 400000 tokens) → sub-manager
│ ├── Specialist A1 (depth 2, budget: 150000 tokens)
│ └── Specialist A2 (depth 2, budget: 150000 tokens)
├── Team Lead B (depth 1, budget: 400000 tokens) → sub-manager
│ └── Specialist B1 (depth 2, budget: 300000 tokens)
└── Worker C (depth 1, budget: 100000 tokens)
Requires budget.max_depth, observability tracing, and circuit breaker.
Submanager auto-promotion (complexity-scored). A worker is promoted to a
submanager only when its subtask passes a deterministic complexity scorer
(subtask description length, keywords like research, validate, design,
declared deliverable count, priority). Subtasks scoring ≥ 3 are promoted,
with bonuses for collective fan-outs. A hard 50% promotion cap prevents a
single iteration from blowing the parent budget on submanagers. This makes
the submanager decision a property of the work itself rather than a manual
flag, while still bounded by the budget envelope.
Budget reservation model. Allocation uses pre-charge reservation:
allocate_child(fraction) immediately books capacity against the parent's
remaining pool, so three concurrent siblings each asking for 30% see a
correctly shrinking pool and the no-over-commit invariant
sum(children) + self <= allocation is maintained even under parallel
spawning. When a sub-run finishes, reclaim_child() refunds any unused
capacity back to the parent. Combined with max_depth and the circuit
breaker, this gives A4 its strong termination guarantees.
Worker stall detection. In addition to manager-level stall detection,
the runtime detects when a worker emits 3 identical tool calls in a row
(stuck retry loop on a broken tool). It then forces a final completion
round with tool_choice="none", allowing the worker to gracefully return
a result instead of looping until budget exhaustion.
Cluster visualization. The UI graph view renders each sub-run as a nested cluster container, color-coded by recursion depth (violet → pink → cyan → amber), so it is visually obvious which submanager spawned which children and how much budget each cluster reserved.
Autonomy level: A4 Self-Organizing