Skip to content

Latest commit

 

History

History
809 lines (652 loc) · 30.5 KB

ORCHESTRATION_ENGINES.md

File metadata and controls

809 lines (652 loc) · 30.5 KB

AWP Orchestration Engines

See also — Parent: docs/README.md, orchestration.md (Layer 5) · Autonomy mapping: compliance.md — dag engine covers A0–A1; delegation_loop engine covers A2–A4 · Delegation-loop concepts: manager-intelligence.md (PLAN/DELEGATE/COMPLETE decisions), critique.md (per-worker repair), runtime.md (budget envelope, completion gate chain, R35 repair fixpoint) · DAG-engine concepts: memory.md (share_output), validation.md (graph/cycle rules) · Observability: observability.md · Spec: spec/versions/1.0/layers/05-orchestration.md

Mental Model

AWP ships two orchestration engines because predictable pipelines and open-ended exploration have fundamentally different control structures, and forcing both into a single abstraction makes both worse.

  • The DAG engine trusts the workflow author to know the steps. Edges are explicit, the graph is static, and the runtime just walks it in topological order. Cost is N agents * cost_per_agent. Auditing is reading a YAML file.
  • The Delegation Loop engine trusts a manager LLM to discover the steps from the task itself, but bounds that trust with a deterministic safety envelope (budgets, forbidden tools, sandbox limits, reservation-based budget arithmetic). Cost is bounded by budget.max_*. Auditing is reading a per-iteration audit trail on disk.

Both engines share everything else: the agent contract (R17 confidence output), the tool registry, the memory tiers, the validators, the observability layer. Choose by the structure of the work, not by autonomy level for its own sake — though as a heuristic, A0-A1 is DAG territory and A2-A4 is delegation-loop territory.

The two engines also compose: a DAG node can be a delegation loop, giving you a predictable outer pipeline with a flexible inner step. See Pattern 4.

Related reading: orchestration.md (DAG field reference + delegation-loop config), manifest.md (envelope declaration), manager-intelligence.md (auto-promotion logic), critique.md (reflective repair loop), runtime-tool-generation.md (B1-B6 tool pipeline).

Each engine answers the same question — "In what order and under what conditions do agents run?" — but with fundamentally different philosophies.

orchestration_engines diagram


Engine 1: DAG (Directed Acyclic Graph)

Status: Available since AWP 1.0.0 Best for: Predictable pipelines with known steps

How It Works

You define a static graph of agents and their dependencies in workflow.awp.yaml. The runtime topologically sorts the graph into execution levels and runs agents in order, respecting dependencies.

orchestration:
  engine: dag
  graph:
    - id: planner
      agent: planner
      depends_on: []
    - id: researcher
      agent: researcher
      depends_on: [planner]
    - id: writer
      agent: writer
      depends_on: [researcher]

Execution Model

orchestration_engines diagram

Agents at the same level with no mutual dependencies run in parallel.

Features

Feature Support
Sequential execution Yes
Parallel execution Yes (same-level, independent agents)
Conditional execution Yes (when expressions)
State sharing Yes (full, selective, isolated)
Retry with backoff Yes
Fan-out Yes (via FanOutConfig)
Loops Yes (per-agent LoopConfig)
Tools Yes (full MCP tool registry)
Code Mode Yes
Dynamic Tool Creation Yes

When to Use DAG

  • The number of steps is known before execution
  • Each agent's role is fixed and well-defined
  • Data flows in one direction (pipeline pattern)
  • You need maximum predictability and auditability
  • Budget planning is straightforward (N agents × cost per agent)

Configuration Reference

orchestration:
  engine: dag
  execution:
    mode: parallel          # sequential | parallel | conditional
    timeout:
      per_agent: 120        # seconds
      total: 300
    max_parallel_agents: 4
    error_handling:
      default: continue     # continue | stop | retry
      max_retries: 1
      retry_delay: 2.0
  graph:
    - id: agent_name
      agent: agent_dir_name
      enabled: true
      depends_on: [other_agent]
      share_output: [field1, field2]
      on_failure: continue  # continue | skip | abort
      when: "state.other_agent.score > 0.5"
      timeout: 60
      retry: 2

Engine 2: Delegation Loop

Status: Available since AWP 1.1.0 Best for: Open-ended tasks where the steps emerge during execution

How It Works

A manager agent receives the task and dynamically decides what to do:

  1. DELEGATE — Generate instructions, skills, and tool configs for worker agents
  2. COMPLETE — The task is done, return the final result
  3. FAIL — The task cannot be completed

Workers are ephemeral — they don't exist as static agent.awp.yaml files. The manager creates them at runtime by generating a Delegation Envelope containing everything the worker needs: instructions, domain knowledge (skills), allowed tools, and output contract.

orchestration_engines diagram

The Delegation Envelope

The manager generates this for each worker:

{
  "worker_id": "market_researcher",
  "instructions": "Research current market trends for electric vehicles...",
  "skills": [
    "## Market Analysis\n- Focus on TAM/SAM/SOM...\n- Use Porter's Five Forces..."
  ],
  "tools_allowed": ["web.search", "file.read"],
  "output_contract": {
    "required_fields": ["trends", "market_size", "confidence"],
    "description": "Market research findings"
  },
  "codemode": {
    "enabled": false,
    "tool_creation": false
  }
}

The manager dynamically generates:

  • Instructions — What the worker should do
  • Skills — Domain knowledge as Markdown, injected into the worker's prompt
  • Tool configuration — Which MCP tools the worker can use
  • Output contract — What the worker must return

Budget System

Instead of a simple depth limit, the delegation loop uses a budget system that tracks resource consumption across the entire execution tree:

budget:
  max_loops: 20              # Maximum manager iterations
  max_total_workers: 30      # Total workers across all iterations
  max_total_tokens: 1000000  # LLM token limit
  max_wall_time: 600         # Wall clock seconds
  max_tool_calls: 200        # Total tool invocations
  max_depth: 2               # Safety limit for recursive sub-delegation (default 2)
  max_concurrent_submanagers: 3     # Max submanagers running at once (default 3)
  max_total_submanagers_per_run: 6  # Hard cap on submanagers per run (default 6)

The budget enforces a hard stop when any resource limit is hit. This prevents runaway loops and guarantees cost predictability.

When workers sub-delegate (recursive delegation), they must allocate portions of their budget to sub-workers. The invariant sum(children) + self <= allocation is enforced deterministically, not by the LLM.

Safety Envelope

The manager controls worker instructions, skills, and tools — but it cannot override security-critical parameters:

worker_policy:
  enforced:                          # Manager CANNOT change these
    sandbox:
      type: subprocess
      max_memory_mb: 512
      max_cpu_seconds: 30
      network: false
    codemode:
      max_tools_per_worker: 10
    rate_limiting:
      max_llm_calls_per_minute: 30
    forbidden_tools:
      - "shell.execute"

  manager_controlled:                # Manager CAN set these
    - instructions
    - skills
    - tools_allowed
    - output_contract
    - codemode.enabled
    - codemode.tool_creation
    - temperature                    # Per-worker LLM temperature

The model selection is also outside the manager's control — it's set by the user via --manager-model and --worker-model CLI flags. However, the manager can set temperature per worker in the delegation envelope to control creativity (e.g., low for analysis, high for brainstorming). If omitted, defaults to 0.2.

Two-Tier Validation

Every worker result passes through two validation stages:

Tier 1: Deterministic (always runs, free)

  • Is the result a valid JSON object?
  • Does it contain the required confidence field?
  • Is confidence in range [0.0, 1.0]?
  • Is the budget still within limits?

Tier 2: LLM Semantic (conditional, costs tokens)

  • Does the result actually address the task?
  • Is the confidence score realistic?
  • Does it contradict previous findings?

The LLM validator runs on the worker model (not the expensive manager model) and can be skipped when confidence is very high or budget is low.

Stall Detection

The loop automatically detects when it's making no progress:

termination:
  enabled: true
  window: 3                  # Compare last 3 iterations
  min_confidence_delta: 0.05 # Must improve by at least 0.05
  action: warn_then_stop     # warn once, stop on second stall

If confidence doesn't improve over window iterations, the system:

  1. First occurrence: warns the manager ("confidence is stagnating")
  2. Second occurrence: terminates the loop with partial results

Context Budget & Smart Spillover

Worker results can be large (DataFrames, analysis outputs, etc.). The context budget controls how much data is inlined into worker prompts vs spilled to files:

context_budget:
  total_chars: 64000    # Total character budget for all context (~16K tokens)
  min_per_entry: 4000   # Floor per entry even with many workers
  preview_chars: 2000   # Preview length for spilled results

Auto-detect (default): The total_chars budget is divided equally among the number of context entries. Each entry gets at least min_per_entry characters.

Spillover behavior:

  • Results under budget: inlined fully as JSON in the worker prompt
  • Results over budget: written to workspace/context/<worker_id>.json, with a truncated preview and file path shown in the prompt
  • Workers can read the full spilled file via code.execute: json.load(open(_workspace_dir + "/context/worker_1.json"))

Input Registry: Workers automatically receive a listing of all available data files in workspace/inputs/ and workspace/context/, including:

  • File name, size, and access path
  • Schema preview for CSV files (column names, first 3 rows, row count)
  • Structure preview for JSON files (top-level keys, array sizes)
  • Read hints for binary formats (Parquet, Feather, etc.)

This is configured at the orchestration level (context_budget field) and applies to both DAG and delegation loop engines.

Rolling Summary

To prevent context window overflow, the loop maintains a rolling summary:

  • Last N iterations: Full results (configurable via full_results_window)
  • Older iterations: Summarized as key findings + confidence
  • On disk: ROLLING_SUMMARY.md (human-readable) + rolling_summary.json (machine-readable)

The manager receives the rolling summary in every iteration, keeping its context size constant regardless of how many iterations have run.

Logging Structure

Every run produces a complete audit trail on disk:

workspace/runs/{run_id}/
├── RUN_SUMMARY.md                 # Human-readable overview
├── run_manifest.json              # Machine-readable config
├── run_completion.json            # Final status and budget
│
├── iterations/
│   ├── 001/
│   │   ├── ITERATION_SUMMARY.md   # What happened
│   │   ├── manager_decision.json  # Manager's raw output
│   │   ├── budget_snapshot.json   # Budget after this iteration
│   │   ├── validation.json        # Validation results
│   │   └── delegations/
│   │       ├── worker_a/
│   │       │   ├── envelope.json  # What the worker received
│   │       │   ├── result.json    # What the worker returned
│   │       │   ├── RESULT.md      # Human-readable result
│   │       │   └── generated_skills/
│   │       │       └── skill_0.md
│   │       └── worker_b/
│   │           └── ...
│   └── 002/
│       └── ...
│
├── history/
│   ├── ROLLING_SUMMARY.md         # Current summary
│   └── rolling_summary.json       # Machine-readable history
│
└── artifacts/
    ├── skills/                    # All generated skills
    └── tools/                     # All generated tools

Both JSON (for machines/agents) and Markdown (for humans/debugging) are generated by default. Use logging.format: json or logging.format: md for single-format.

Model Configuration

The user controls which models are used:

# Manager gets the strong model, workers get the fast model
awp run my-workflow/ \
  --task "Analyze market trends" \
  --manager-model openrouter/anthropic/claude-opus-4 \
  --worker-model openrouter/anthropic/claude-sonnet-4

Resolution order:

--manager-model  →  workflow YAML models.manager  →  LLM_MODEL env  →  Error
--worker-model   →  workflow YAML models.worker   →  manager model  →  Error

The manager cannot override the worker model. This prevents a hallucinating manager from upgrading workers to expensive models.

Recursive Sub-Delegation

Workers can act as managers and spawn their own sub-workers:

Manager (depth 0)
├── Worker A (depth 1) → becomes sub-manager
│   ├── Sub-Worker A1 (depth 2)
│   └── Sub-Worker A2 (depth 2)
├── Worker B (depth 1)
└── Worker C (depth 1)

This is controlled by the budget system, not just depth limits. A worker that sub-delegates must allocate portions of its budget to sub-workers.

Configuration Reference

orchestration:
  engine: delegation_loop

  delegation_loop:
    manager: agents/manager        # Agent directory for the manager

    models:
      manager: null                # CLI → YAML → LLM_MODEL
      worker: null                 # CLI → YAML → manager model

    worker_policy:
      enforced:
        sandbox:
          type: subprocess
          max_memory_mb: 512
          max_cpu_seconds: 30
          network: false
        codemode:
          max_tools_per_worker: 10
        rate_limiting:
          max_llm_calls_per_minute: 30
        forbidden_tools:
          - "shell.execute"
      manager_controlled:
        - instructions
        - skills
        - tools_allowed
        - output_contract
        - codemode.enabled
        - codemode.tool_creation
        - temperature

    budget:
      max_loops: 20
      max_total_workers: 30
      max_total_tokens: 1000000
      max_wall_time: 600
      max_tool_calls: 200

    context_budget:
      total_chars: 64000   # auto-divided among workers
      min_per_entry: 4000  # floor per entry
      preview_chars: 2000  # preview for spilled results
      max_depth: 5

    termination:
      enabled: true
      window: 3
      min_confidence_delta: 0.05
      action: warn_then_stop

    validation:
      deterministic:
        always: true
        checks: [schema, required_fields, confidence, budget]
      llm:
        enabled: true
        skip_when_confidence_above: 0.95
        skip_when_budget_remaining_below: 0.1

    history:
      rolling_summary: true
      full_results_window: 3
      persist_to_disk: true

    logging:
      format: dual              # dual | json | md
      persist_artifacts: true

Per-worker Executor Lifecycle

Within a delegation-loop run, every code.execute call inside a single worker targets the same warm Python subprocess — a long-running child managed by PersistentExecutor. Variables, imports, and helper functions defined in one call remain live in the next; heavy imports (numpy, pandas, matplotlib) are paid once per worker, not once per call. Implementation details that matter for reasoning about run behavior:

  • Merged stderr/stdout. The warm child merges stderr into stdout on the pipe level, so a chatty child process cannot deadlock on a full stderr buffer. Real tracebacks are still returned inside the JSON result envelope.
  • Deadline-accurate reads. Reads happen on the raw file descriptor with select() and a private byte buffer; the configured per-call timeout is honored even if the child produces no output.
  • Auto-restart with history replay. If the warm subprocess dies between calls, a fresh one is spawned and the recorded history (up to 100 entries / 1 MB of code text) is silently replayed so the caller sees a warm namespace again. Timeouts (status 408) are NOT recorded — replaying a hang would deadlock the restart. If warm execution fails at the protocol level, the call falls back to a one-shot subprocess.run (cold-start) and the loop continues.

Across workers, the runner maintains a per-logical-worker executor registry. A logical id is derived by stripping repair / retry suffixes (_repair, _retry, _vN, _strict, _final, _runN, _subtask_N) to a fixpoint, case-insensitively. The first worker of a run claims the shared executor; brand-new logical ids spawn their own executors. When a repair variant of the same worker is dispatched (compose_worker_repair, plot_runner_v2, etc.), the runtime re-enters the prior worker's executor — the namespace, imports, and partial state from the first attempt are still live — and prepends a ## REPAIR MODE (Continuing Prior Work) block to the worker system prompt so the LLM treats the attempt as an incremental fix rather than a cold start.

The manager can opt out and force a clean slate via the delegation-envelope field fresh_worker: true. This field is part of the envelope schema, not a user-facing config flag. All per-worker executors are cleaned up in the run finalizer; cleanup errors are logged but never propagate.

Authoritative code: packages/awp-runtime/src/awp/runtime/persistent_executor.py, packages/awp-runtime/src/awp/runtime/delegation_loop_runner.py (logical_worker_id, _get_executor_for_worker, REPAIR MODE marker).

Auto-Emergent Tools

Managers and workers rarely call tool.create voluntarily, even when they repeatedly ask code.execute to run the same shape of code with different literal values. The delegation-loop runtime closes that gap with a deterministic tool inducer that watches every code.execute call and computes an AST-skeleton signature with literals normalized to <const> and identifiers renamed to <vK> by first-occurrence index. Structurally identical snippets share the same signature regardless of paths, filenames, or variable names they plug in.

The inducer counts distinct worker ids per signature. When N=3 different workers have executed the same shape, the runtime:

  1. Lifts the varying literal slots to parameters.
  2. Emits a generalized Python tool and registers it through DynamicToolFactory.
  3. Persists it at shared/dynamic_tools/dynamic.induced_<hash6>.json so later workers discover it via tool.list.

N=3 is a hardcoded constant — no new user-facing config flag was introduced. If the varying slots cannot be cleanly parameterized (nested expressions rather than plain constants), the inducer logs an INFO line and skips synthesis rather than emit a broken tool. Induction never rewrites or delays the observed code.execute call; it is pure addition. Every emission is listed in run_completion.json.induced_tools.

Authoritative code: packages/awp-runtime/src/awp/runtime/tool_inducer.py.

Atomicity Advisory

Before the manager plans the next iteration, each pending subtask is scored on a deterministic 0.0-1.0 axis:

  • 1.0 → atomic; prefer a single flat ephemeral worker.
  • 0.0 → composite; a sub-manager with further decomposition is likely warranted.

The score is computed from description features only — atomic-keyword count (read, compute, verify, ...), composition-keyword count (plan, coordinate, orchestrate, synthesize, ...), declared required-outputs count, numbered-step count in the description, and log of word count. The exact formula and weights live in packages/awp-runtime/src/awp/runtime/atomicity.py.

The score is advisory only. It is injected into the manager planning prompt as a hint the LLM may weigh alongside its own reasoning. The runtime MUST NOT use the score to block, override, or auto-promote a manager delegation decision — that would violate AWP's A3/A4 autonomy semantics (the manager is the authority on delegation shape). The value is surfaced for observability and tuning, not for control flow.

Preemptive Wall-Time Watchdog (Framework-Fix δ-1)

The cooperative budget check at the top of each manager iteration is insufficient when a single iteration blocks indefinitely — e.g. when a code.execute call deadlocks or an LLM client stalls. A daemon thread named awp-walltime-watchdog is started in DelegationLoopRunner.run() (only at depth == 0) and polls absolute wall-time every 30 s. On breach of max_wall_time, it sends SIGTERM to its own process, which the existing signal handler converts into a clean status=partial exit with reason=max_wall_time and a written run_completion.json. If the process is still alive 30 s after SIGTERM, the watchdog escalates to SIGKILL. The watchdog is unaffected by main-thread deadlocks — it is the structural guarantee that max_wall_time is honored under every failure mode.

Embedded-mode guard (allow_process_kill=False): When the runner is constructed via AgentWorkflow(allow_process_kill=False) — the path the Studio UI server uses so the runner lives as a thread inside a long-lived web process — the watchdog still hard-breaches on budget overrun but swaps the SIGTERM/SIGKILL sequence for LLMClient.close_all(). All in-flight HTTP calls raise ReadError; the runner's except/finally blocks run as in the CLI path and produce the same status=partial/reason=max_wall_time terminal — without killing the host server. CLI invocations keep the original signal path.

Hard Per-Call Executor Timeout Killer (Framework-Fix δ-2)

Inside PersistentExecutor.execute() a threading.Timer is armed for effective_timeout + 5 s before the call enters the warm-subprocess read path. When the timer fires, a separate thread forcibly kills the subprocess and sets _needs_replay=True + _hard_killed=True. This is the second defense line behind α-1's select-based deadline. If the main read thread somehow never wakes (stuck os.read, kernel-level pipe issue), the timer still kills the child. The caller then gets a structured cold-fallback response rather than an indefinite wait.

Pipe-Output Cap in Worker Script (Framework-Fix δ-3)

The warm subprocess is driven by _WORKER_SCRIPT which captures stdout and stderr into io.StringIO buffers during exec(code, _ns). Before serialising the per-call JSON result, both buffers are now capped at 2 MB UTF-8 bytes. Oversized output is truncated and a ...[truncated: original N bytes] marker is appended. Prior to this cap, a print() of a multi-megabyte blob inside the worker could overflow the 64 KB Linux pipe buffer while the parent was momentarily busy — classic producer-consumer deadlock. The cap eliminates that class of hang at the source, in the child, before the byte ever touches the pipe.


Composing Engines: DAG + Delegation Loop

The two engines can be composed. A DAG node can be a delegation loop:

orchestration:
  engine: dag
  graph:
    - id: gather_requirements
      agent: requirements_gatherer

    - id: dynamic_analysis
      type: delegation_loop
      config:
        manager: agents/analysis_manager
        budget: {max_loops: 10, max_total_workers: 15}
      depends_on: [gather_requirements]

    - id: write_report
      agent: report_writer
      depends_on: [dynamic_analysis]

In this pattern:

  1. gather_requirements runs first (DAG agent)
  2. dynamic_analysis runs as a delegation loop (multiple iterations)
  3. write_report runs after the loop completes (DAG agent)

This gives you the predictability of a DAG for the overall flow, with the flexibility of a delegation loop for the complex middle step.


Choosing an Engine

Factor DAG Delegation Loop
Steps known upfront Yes No
Cost predictability High (N agents) Medium (budget-bounded)
Adaptability Low High
Debugging Simple (trace the graph) Rich (full audit trail on disk)
Best for Pipelines, ETL, fixed workflows Research, analysis, open-ended tasks
Agent definition Static (agent.awp.yaml) Manager: static, Workers: dynamic
LLM calls 1 per agent 1 per manager iteration + 1 per worker
Parallelism Same-level agents Fan-out workers per iteration

Rules of Thumb

  1. Start with DAG. If you can enumerate the steps, use a DAG.
  2. Use Delegation Loop when the number of steps depends on the task content.
  3. Compose both when you have a predictable outer flow with an adaptive inner step.
  4. Budget aggressively. Set conservative limits and increase if needed.
  5. Enable stall detection. It's your safety net against infinite loops.

Design Patterns

Pattern 1: Pipeline (DAG)

A linear chain of agents where each builds on the previous output. Classic ETL or research-then-write flow.

orchestration:
  engine: dag
  graph:
    - id: researcher
      agent: researcher
      depends_on: []
      share_output: [findings]
    - id: writer
      agent: writer
      depends_on: [researcher]
      share_output: [article]

Autonomy level: A1 Adaptive

Pattern 2: Fan-Out / Fan-In (DAG)

Multiple agents run in parallel, then a synthesizer combines results.

orchestration:
  engine: dag
  graph:
    - id: analyst_a
      agent: analyst
      depends_on: []
      share_output: [analysis]
    - id: analyst_b
      agent: analyst
      depends_on: []
      share_output: [analysis]
    - id: synthesizer
      agent: synthesizer
      depends_on: [analyst_a, analyst_b]

Autonomy level: A1 Adaptive

Pattern 3: Dynamic Delegation (Delegation Loop)

A manager decomposes a task into subtasks and spawns workers dynamically. The number and type of workers depend on the task.

orchestration:
  engine: delegation_loop
  delegation_loop:
    manager: agents/project_manager
    budget:
      max_loops: 10
      max_total_workers: 20
    termination:
      enabled: true
      window: 3
      min_confidence_delta: 0.05

Autonomy level: A2 Delegating

Pattern 4: DAG with Delegation Loop Inner Step (Composed)

A predictable outer pipeline wraps a dynamic inner step.

orchestration:
  engine: dag
  graph:
    - id: gather_requirements
      agent: requirements_gatherer
    - id: dynamic_analysis
      type: delegation_loop
      config:
        manager: agents/analysis_manager
        budget: { max_loops: 10, max_total_workers: 15 }
      depends_on: [gather_requirements]
    - id: write_report
      agent: report_writer
      depends_on: [dynamic_analysis]

Autonomy level: A2 Delegating

Pattern 5: Self-Tooling Workers (Delegation Loop + CodeMode)

Workers create MCP tools at runtime to handle unfamiliar APIs or data formats.

orchestration:
  engine: delegation_loop
  delegation_loop:
    manager: agents/integration_manager
    budget:
      max_loops: 15
      max_total_workers: 20
      max_tool_calls: 300
    worker_policy:
      enforced:
        codemode:
          max_tools_per_worker: 5
        forbidden_tools: ["shell.execute"]
      manager_controlled:
        - instructions
        - skills
        - tools_allowed
        - codemode.enabled
        - codemode.tool_creation
        - temperature

Autonomy level: A3 Self-Tooling

Pattern 6: Recursive Delegation (Delegation Loop, A4)

Workers act as sub-managers, spawning their own workers. Budget flows down the hierarchy with the invariant sum(children) + self <= allocation.

Manager (depth 0, budget: 1000000 tokens)
├── Team Lead A (depth 1, budget: 400000 tokens) → sub-manager
│   ├── Specialist A1 (depth 2, budget: 150000 tokens)
│   └── Specialist A2 (depth 2, budget: 150000 tokens)
├── Team Lead B (depth 1, budget: 400000 tokens) → sub-manager
│   └── Specialist B1 (depth 2, budget: 300000 tokens)
└── Worker C (depth 1, budget: 100000 tokens)

Requires budget.max_depth, observability tracing, and circuit breaker.

Submanager auto-promotion (complexity-scored). A worker is promoted to a submanager only when its subtask passes a deterministic complexity scorer (subtask description length, keywords like research, validate, design, declared deliverable count, priority). Subtasks scoring ≥ 3 are promoted, with bonuses for collective fan-outs. A hard 50% promotion cap prevents a single iteration from blowing the parent budget on submanagers. This makes the submanager decision a property of the work itself rather than a manual flag, while still bounded by the budget envelope.

Budget reservation model. Allocation uses pre-charge reservation: allocate_child(fraction) immediately books capacity against the parent's remaining pool, so three concurrent siblings each asking for 30% see a correctly shrinking pool and the no-over-commit invariant sum(children) + self <= allocation is maintained even under parallel spawning. When a sub-run finishes, reclaim_child() refunds any unused capacity back to the parent. Combined with max_depth and the circuit breaker, this gives A4 its strong termination guarantees.

Worker stall detection. In addition to manager-level stall detection, the runtime detects when a worker emits 3 identical tool calls in a row (stuck retry loop on a broken tool). It then forces a final completion round with tool_choice="none", allowing the worker to gracefully return a result instead of looping until budget exhaustion.

Cluster visualization. The UI graph view renders each sub-run as a nested cluster container, color-coded by recursion depth (violet → pink → cyan → amber), so it is visually obvious which submanager spawned which children and how much budget each cluster reserved.

Autonomy level: A4 Self-Organizing