Skip to content

Latest commit

 

History

History
319 lines (282 loc) · 19.3 KB

ARCHITECTURE.md

File metadata and controls

319 lines (282 loc) · 19.3 KB

Architecture

coop is a Rust CLI that orchestrates isolated VM environments for running AI coding agents (Claude Code, Codex, Grok Build). It manages the full VM lifecycle — setup, start, shell, stop, destroy, status, logs — behind two platform backends:

  • Linux — Firecracker microVMs on KVM.
  • macOS — Lima VMs on Apple Virtualization.framework (limactl).

This document maps the modules, the two-backend design, the data flow from host to guest, and the architectural invariants. For the security view of the same system, see trust-model.md; for Rust conventions, see code-style.md.

Layout

coop/
├── src/
│   ├── main.rs             # thin binary shim → coop::run()
│   ├── lib.rs              # clap CLI definition + central command dispatcher
│   ├── backend.rs          # VmBackend trait, PlatformBackend alias, shared guest ops
│   ├── vm.rs               # Firecracker process management (typestate machine)
│   ├── lima.rs             # macOS/Lima backend implementation
│   ├── setup.rs            # Firecracker host setup + golden-image builder
│   ├── firecracker_install.rs # pinned, root-owned Firecracker/jailer pair
│   ├── firecracker_service.rs # systemd VMM service identity and lifecycle
│   ├── setup_tool.rs       # Firecracker setup-tool roles and launch policy
│   ├── network.rs          # Firecracker TAP/bridge/NAT networking
│   ├── config.rs           # config model + loading (the type-safe core)
│   ├── workspace.rs        # workspace sync (rsync) + ~/.ssh/config injection
│   ├── guest.rs            # guest profiles, required binaries, baked package lists
│   ├── guest_env_state.rs  # persisted guest env vars
│   ├── host_tool.rs        # typed trusted host-tool resolution + launch context
│   ├── openssh.rs          # isolated coop-managed SSH/SCP/rsync boundary
│   ├── vsock_proxy.rs      # bounded Firecracker CONNECT handshake + byte relay
│   ├── proxy.rs            # host proxy processes and per-provider SSH reverse tunnels
│   ├── web_editor.rs       # on-demand editor sessions, private IPC and SSH orchestration
│   ├── proxy_state.rs      # persisted per-instance credential overrides
│   ├── model_state.rs      # per-instance local/remote model routing
│   ├── secret_store.rs     # pluggable secret backends (keychain/op/secret-tool/file)
│   ├── github_pat.rs       # `coop github` PAT wizard (setup/rotate/status/forget)
│   ├── github_assignment.rs # persistent VM selection of an existing PAT entry
│   ├── github_repo.rs      # RepoSlug + repo-URL parsing
│   ├── github_submodules.rs# submodule discovery for PAT scoping
│   ├── pat_prompt.rs       # interactive pre-start PAT prompt
│   ├── ssh.rs              # interactive SSH launching
│   ├── remote_command.rs   # injection-safe remote shell-command builder
│   ├── cmd.rs              # host std::process::Command wrapper (redaction, sudo)
│   ├── process_output.rs   # bounded concurrent capture + transient process groups
│   ├── port_forward.rs     # ssh -L port forwards + state
│   ├── signal.rs           # SIGINT/SIGTERM cooperative cancellation
│   ├── paths.rs            # GuestPath / HostPath type split
│   ├── fs_util.rs          # private directories/ACLs, atomic file writes, file locks
│   ├── private_storage.rs  # managed state and disk permission repair
│   ├── sha256_hash.rs      # Sha256Hash newtype
│   ├── naming.rs           # safe-name character class
│   ├── completions.rs      # dynamic shell registration + candidates
│   ├── prompt.rs           # TTY prompts
│   ├── update.rs           # `coop update` self-update + background notifier
│   └── commands/           # one module per command domain (see below)
├── coop-proxy/             # separate binary crate: credential injection, policy, TLS, jail
├── scripts/guest/          # guest-image provisioning scripts (embedded at build)
├── guest/init.sh           # guest first-boot init
├── tests/                  # integration test scripts (see AGENTS.md "Before committing")
├── fuzz/                   # cargo-fuzz targets (own workspace)
└── docs/                   # this tree

The main CLI logic lives in the library crate (src/lib.rs); main.rs is a thin shim calling coop::run(). Its unit and mutation tests target the library. The credential proxy is a separate binary crate in the same Cargo workspace; --workspace builds and tests both crates. Plain cargo build and cargo test select only the root coop package.

The two-backend design

The central abstraction is the backend::VmBackend trait — every VM operation (setup, create_and_start, start_existing, stop, destroy_instance, resize_disk, commit_disk, status_named, stream_logs, ssh_target, …) goes through it. Two implementations exist:

  • FirecrackerBackend — #[cfg(not(target_os = "macos"))]; delegates to setup, vm::FirecrackerVm, and network.
  • LimaBackend — #[cfg(target_os = "macos")]; delegates to lima.

Backend selection is compile-time, not runtime. backend::PlatformBackend is a type alias resolved by #[cfg] — LimaBackend on macOS, FirecrackerBackend elsewhere. There is no runtime backend enum and no dispatch cost. (Note: the Backend enum in secret_store.rs is unrelated — it names secret-storage backends.)

Everything above the trait is backend-shared: the entire "shared guest operations" surface in backend.rs (env/secret forwarding, agent bootstrap, Claude/Codex/Grok config injection, git-repo cloning), plus workspace.rs, ssh.rs, config.rs, and the commands/ handlers. Shared behavior applies to both backends. Known intentional divergences:

Aspect Firecracker Lima
guest_host_address TAP gateway (network.host_ip) host.lima.internal
mounts_are_live false (one-time rsync sync) true (virtiofs)
ssh_target legacy TCP from guest_ip + ssh_port, or persisted vsock socket + guest port queried from limactl (per-boot forwarded port)
copied workspace (--copy) one-time rsync over SSH one-time rsync over SSH
project and extra mounts one-time rsync over SSH live virtiofs mounts

Typestate invariants

Two lifecycle machines are encoded in the type system rather than in runtime flags — this is a load-bearing design choice (see code-style.md):

  • RunningInstance / StoppedInstance (backend.rs) have private fields. as_running records a point-in-time running observation; as_stopped and a successful stop mint StoppedInstance only after confirming exit. StoppedInstance holds the per-instance operation lock through cleanup and stopped-only disk mutations. Lima only mints it for confirmed Stopped; absent, broken, unknown, and failed probes stay distinct. A machine-resource update consumes the stopped token because Lima validates it by booting the VM and Firecracker may boot when --start is requested; the returned operation guard keeps the lock without claiming the VM is still stopped.
  • FirecrackerVm<Configured> / FirecrackerVm<Running> (vm.rs) gate start()/stop() transitions at compile time.
  • Setup preflight rejects explicit inputs unsupported by the selected backend before config loading at the CLI boundary and repeats that capability check at the backend boundary. boot_preflight(cfg) then runs cfg.validate() before backend setup or boot work, so no VM starts on an invalid config.

backend::SshTarget keeps backend identity separate from its closed transport choice. TCP targets carry a validated hostname and port. The Firecracker-vsock variant carries the trusted per-instance Unix-socket path and guest port, and renders the same Rust __vsock-proxy helper for SSH, SCP, rsync, forwarding, and generated SSH configuration. The helper bounds and validates Firecracker's text CONNECT handshake before relaying raw bytes. New Firecracker instances persist the configured FirecrackerControlMode; the opt-in vsock mode is then used for readiness, shutdown, and every shared SSH consumer. Older state without the tag maps to LegacyTcp in memory without an eager rewrite. Lima remains TCP-based.

Firecracker instances also persist an open or isolated network mode. The isolated representation uses the composite vsock_isolated value in the existing control field so supported prior releases reject it instead of ignoring a new field and booting with a NIC. In memory it normalizes to vsock control plus an isolated network policy. Firecracker config generation emits an empty network-interfaces list and backend startup skips TAP setup; every shared SSH consumer continues through the same vsock target.

An instance operation lock is held from the stopped-state probe through disk resize, commit, or restore. Start, stop, and destroy acquire the same bounded lock. Its file is a sibling of the instance directory, so directory removal does not replace the lock inode. Allocation releases its directory lock before any instance operation lock is acquired.

Command dispatch

lib.rs:run() is the entrypoint. Order matters:

  1. Emit dynamic shell completions (before arg parsing) if requested.
  2. Parse Cli (clap derive).
  3. Reject backend-unsupported explicit setup inputs before tracing, config loading, or other setup side effects.
  4. Init tracing (→ stderr) and handle commands that must work without a loaded config — Completions, Init, Update, Uninstall — first.
  5. config::CoopConfig::load; on Linux, preflight Firecracker setup tools before managed-storage repair. Non-setup commands prepare storage here. Print the update notice; run its background check except during Linux setup.
  6. Construct backend::PlatformBackend::new().
  7. match each Commands variant to a commands::cmd_* handler. Lifecycle handlers call cfg.validate_and_warn()? before VM work; query commands (list/status/logs) skip it.

The commands/ submodules own the domains: lifecycle.rs (up/start/shell/ exec/stop/destroy/status/list/resize/commit/restore), quickstart.rs, profiles.rs (+ images), agent.rs (coop agent update), model.rs (coop model), proxy.rs (coop proxy), github.rs, admin.rs (init/validate/uninstall), and json.rs (machine-readable --json output types). commands/mod.rs re-exports the dispatch surface and holds cross-domain helpers (merge_runtime_guest_env, purge_all_data).

Data flow: host → guest

The lifecycle is setup → up/start → shell → stop → destroy. A first boot (lifecycle.rs:start_instance, BootMode::FirstBoot) runs roughly:

  1. Resolve the target repo/workspace and optionally prompt for a GitHub PAT (pat_prompt::maybe_prompt).
  2. Ports — merge forward-port specs and fail fast on host-port collisions (port_forward::check_host_port_collisions). Stage explicit guest_files sources in private host temporary directories, resolving permitted symlinks and rejecting invalid source trees before boot or disk replacement.
  3. Boot the VM (be.create_and_start), then wait_until_ready (SSH probe with backoff).
  4. Forwards + state — spawn ssh -L forwards, persist ForwardsState, GuestEnvState as JSON sidecars in the instance dir.
  5. Guest files and bootstrap — guest_files.rs uploads the staged copies; scripts/guest/copy-files.sh rejects destination symlinks and live-mount overlaps, then merges private, writable files without deletion. Host staging drops at the end of the operation; guest staging is cleaned after each copy. Then (backend.rs:bootstrap_agents): if a GITHUB_TOKEN is present, gh auth setup-git; then bootstrap_claude / bootstrap_codex inject the allowlisted config files and a managed settings.json, and on first boot install the delta of marketplaces/plugins/MCP servers not already baked into the golden image.
  6. Workspace — --workspace copies via rsync; --git-repo clones inside the guest; mounts are live on Lima and rsync'd on Firecracker. Persist WorkspaceState.
  7. Creation hooks — select a recipe before first boot and persist it in owner-only creation.json. After workspace provisioning, run global post_create in the guest home. Save successful completion atomically. Failures retain the VM and block agent launch; running up and stopped-instance restarts retry the pending hook under the provisioning lock. The instance operation lock covers the initial state read and the final liveness probe, recipe revalidation, and completion write; it is released while the guest hook runs so stop and destroy can proceed. A hook that succeeds but loses the VM or cannot save completion stays pending and may run again on retry. Guest execution uses a private staged script, a PTY-bound supervisor, and a guest lock; cancellation terminates and reaps the supervised process group. Shell and exec remain available for recovery.
  8. post_start hook — runs after creation, workspace, and mount provisioning, with an SSH session prepared after bootstrap to include newly created proxy capability tokens. Hook command failures are warned, not fatal.

Config, secrets, and workspace all cross the host→guest boundary here; the security-relevant details of each crossing are in trust-model.md.

Config model

config::CoopConfig is the type-safe core. Loading reads TOML (or JSON by extension) or defaults, then expands user paths. Value bounds are enforced by newtype constructors (VmMemory has a 128-MiB floor, InstanceIndex is 0..=252, SubnetMask is 0..=32, Hostname/SshUser/EnvVarName validate their character classes), so validate() only checks environmental facts (paths exist, binaries are present). Secrets are stored indirected as cmd: retrieval commands and resolved at VM-start by resolve_cmd_value.

Per-instance runtime state is a set of JSON sidecar files under the instance dir: instance.json, vm_config.json, workspace.json, forwards.json, guest_env.json, model.json, proxy.json, creation.json (selected creation recipe, completion, and deferred startup hook; mode 0600), plus the Firecracker .pid/.socket/.log/vsock files. Allocation refuses an occupied instance path. If any instance directory has unreadable metadata, allocation stops because its network index cannot be trusted; existing healthy instances remain available through normal lookup.

coop update

update.rs self-updates the CLI and its proxy companion: fetch release metadata from the fixed https://api.github.com/repos/trailofbits/coop/releases source, validate the canonical release tag and exact GitHub identity of the platform tarball, SHA256SUMS, and optional attestations.jsonl before download, verify the checksum (mandatory), verify the Sigstore attestation via gh against that bundle, fetching a bundle anonymously from the attestations API by digest when the release publishes no usable one. The installer uses jq for API extraction and retains the authenticated API path when jq is absent. Both clients retain authenticated verification if the anonymous API transfer fails. Verification is skipped only when gh is absent or the explicit test policy disables it. Both verification paths pin the signer to .github/workflows/release.yml at the selected tag, require that source ref, and reject attestations generated on self-hosted runners. Then extract with path-escape-safe tar flags, and atomically rename the new coop-proxy beside the CLI before replacing coop. Each replacement is atomic; the pair is not a single transaction. Official builds cannot replace the metadata origin at runtime. A separately compiled test policy permits only a loopback fixture for the host-only updater integration test. A background notifier checks for new versions on a 24-hour interval (disabled in dev/CI/non-TTY). The full verification chain and its trust properties are documented in trust-model.md.

Guest image

The guest runs from a golden image built once and reused (reflink-cloned per instance on Firecracker). setup.rs builds it via a chroot, running the embedded provisioning scripts in scripts/guest/ (preamble.sh, guest-config.sh, cleanup.sh) plus the profile/package installers referenced from guest.rs. Firecracker uses a minimal CI kernel that is missing several netfilter modules; the resulting workarounds (iptables-legacy, static resolv.conf, DOCKER_INSECURE_NO_IPTABLES_RAW=1) are applied in guest-config.sh and explained in platform-notes.md and trust-model.md.

Architectural invariants

These invariants describe the current architecture:

  1. The VM is the isolation boundary. The guest is untrusted from the host's view; guest-authored data must not escalate into host code execution or filesystem escape. See trust-model.md.
  2. Backend selection is compile-time. PlatformBackend resolves to Firecracker or Lima at compile time; shared code supports both.
  3. State-gated operations use probes and locks. VM operations use RunningInstance/StoppedInstance and boot_preflight; the stopped token holds the operation lock until the disk mutation is complete.
  4. Value invariants live in constructors. Boundaries parse values into newtypes, so downstream code receives values with established invariants.
  5. Secrets use bounded transport and redacted diagnostics. Env/SendEnv/stdin carry most secrets; traces redact them. The macOS security and 1Password op store commands are documented argv exceptions (see trust-model.md).
  6. Remote commands use RemoteCommand::arg; host commands use Cmd::arg. Tainted input is not interpolated into shell command strings.
  7. State is transparent JSON sidecars written atomically (fs_util::atomic_write_*) without relaxing permissions.

The Linux network control plane adds a narrower host-launch invariant on top of item 6. Its sudo, ip, bridge, iptables, and sysctl identities are resolved from fixed absolute system candidates, validated before use, and launched from / with an environment cleared to deterministic locale values. The invocation-selected config may replace any identity with one exact absolute path for non-FHS layouts. An override never falls back; it retains the full root-owned, non-writable path-chain checks but intentionally does not claim package, hash, basename, or command-semantic identity. setup_tap resolves the complete set before its first probe or mutation. Firecracker setup also uses the resolver for its fixed system-tool family: bootstrap tools are checked before storage changes or package installation, and the installed family is checked afterward, before artifact downloads and image construction. The privileged disk helper repeats filesystem-tool resolution at its elevated launch boundary. Other host subprocess families retain their own launch contracts.