coop is a Rust CLI that orchestrates isolated VM environments for running AI
coding agents (Claude Code, Codex, Grok Build). It manages the full VM lifecycle — setup,
start, shell, stop, destroy, status, logs — behind two platform backends:
- Linux — Firecracker microVMs on KVM.
- macOS — Lima VMs on Apple Virtualization.framework (
limactl).
This document maps the modules, the two-backend design, the data flow from host
to guest, and the architectural invariants. For the security view of the same
system, see trust-model.md; for Rust conventions, see
code-style.md.
coop/
├── src/
│ ├── main.rs # thin binary shim → coop::run()
│ ├── lib.rs # clap CLI definition + central command dispatcher
│ ├── backend.rs # VmBackend trait, PlatformBackend alias, shared guest ops
│ ├── vm.rs # Firecracker process management (typestate machine)
│ ├── lima.rs # macOS/Lima backend implementation
│ ├── setup.rs # Firecracker host setup + golden-image builder
│ ├── firecracker_install.rs # pinned, root-owned Firecracker/jailer pair
│ ├── firecracker_service.rs # systemd VMM service identity and lifecycle
│ ├── setup_tool.rs # Firecracker setup-tool roles and launch policy
│ ├── network.rs # Firecracker TAP/bridge/NAT networking
│ ├── config.rs # config model + loading (the type-safe core)
│ ├── workspace.rs # workspace sync (rsync) + ~/.ssh/config injection
│ ├── guest.rs # guest profiles, required binaries, baked package lists
│ ├── guest_env_state.rs # persisted guest env vars
│ ├── host_tool.rs # typed trusted host-tool resolution + launch context
│ ├── openssh.rs # isolated coop-managed SSH/SCP/rsync boundary
│ ├── vsock_proxy.rs # bounded Firecracker CONNECT handshake + byte relay
│ ├── proxy.rs # host proxy processes and per-provider SSH reverse tunnels
│ ├── web_editor.rs # on-demand editor sessions, private IPC and SSH orchestration
│ ├── proxy_state.rs # persisted per-instance credential overrides
│ ├── model_state.rs # per-instance local/remote model routing
│ ├── secret_store.rs # pluggable secret backends (keychain/op/secret-tool/file)
│ ├── github_pat.rs # `coop github` PAT wizard (setup/rotate/status/forget)
│ ├── github_assignment.rs # persistent VM selection of an existing PAT entry
│ ├── github_repo.rs # RepoSlug + repo-URL parsing
│ ├── github_submodules.rs# submodule discovery for PAT scoping
│ ├── pat_prompt.rs # interactive pre-start PAT prompt
│ ├── ssh.rs # interactive SSH launching
│ ├── remote_command.rs # injection-safe remote shell-command builder
│ ├── cmd.rs # host std::process::Command wrapper (redaction, sudo)
│ ├── process_output.rs # bounded concurrent capture + transient process groups
│ ├── port_forward.rs # ssh -L port forwards + state
│ ├── signal.rs # SIGINT/SIGTERM cooperative cancellation
│ ├── paths.rs # GuestPath / HostPath type split
│ ├── fs_util.rs # private directories/ACLs, atomic file writes, file locks
│ ├── private_storage.rs # managed state and disk permission repair
│ ├── sha256_hash.rs # Sha256Hash newtype
│ ├── naming.rs # safe-name character class
│ ├── completions.rs # dynamic shell registration + candidates
│ ├── prompt.rs # TTY prompts
│ ├── update.rs # `coop update` self-update + background notifier
│ └── commands/ # one module per command domain (see below)
├── coop-proxy/ # separate binary crate: credential injection, policy, TLS, jail
├── scripts/guest/ # guest-image provisioning scripts (embedded at build)
├── guest/init.sh # guest first-boot init
├── tests/ # integration test scripts (see AGENTS.md "Before committing")
├── fuzz/ # cargo-fuzz targets (own workspace)
└── docs/ # this tree
The main CLI logic lives in the library crate (src/lib.rs); main.rs
is a thin shim calling coop::run(). Its unit and mutation tests target the
library. The credential proxy is a separate binary crate in the same Cargo
workspace; --workspace builds and tests both crates. Plain cargo build
and cargo test select only the root coop package.
The central abstraction is the backend::VmBackend trait — every VM operation
(setup, create_and_start, start_existing, stop, destroy_instance,
resize_disk, commit_disk, status_named, stream_logs, ssh_target, …) goes
through it. Two implementations exist:
FirecrackerBackend—#[cfg(not(target_os = "macos"))]; delegates tosetup,vm::FirecrackerVm, andnetwork.LimaBackend—#[cfg(target_os = "macos")]; delegates tolima.
Backend selection is compile-time, not runtime. backend::PlatformBackend
is a type alias resolved by #[cfg] — LimaBackend on macOS, FirecrackerBackend
elsewhere. There is no runtime backend enum and no dispatch cost. (Note: the
Backend enum in secret_store.rs is unrelated — it names secret-storage
backends.)
Everything above the trait is backend-shared: the entire "shared guest
operations" surface in backend.rs (env/secret forwarding, agent bootstrap,
Claude/Codex/Grok config injection, git-repo cloning), plus workspace.rs,
ssh.rs, config.rs, and the commands/ handlers. Shared behavior applies
to both backends. Known intentional divergences:
| Aspect | Firecracker | Lima |
|---|---|---|
guest_host_address |
TAP gateway (network.host_ip) |
host.lima.internal |
mounts_are_live |
false (one-time rsync sync) |
true (virtiofs) |
ssh_target |
legacy TCP from guest_ip + ssh_port, or persisted vsock socket + guest port |
queried from limactl (per-boot forwarded port) |
copied workspace (--copy) |
one-time rsync over SSH | one-time rsync over SSH |
| project and extra mounts | one-time rsync over SSH | live virtiofs mounts |
Two lifecycle machines are encoded in the type system rather than in runtime
flags — this is a load-bearing design choice (see
code-style.md):
RunningInstance/StoppedInstance(backend.rs) have private fields.as_runningrecords a point-in-time running observation;as_stoppedand a successfulstopmintStoppedInstanceonly after confirming exit.StoppedInstanceholds the per-instance operation lock through cleanup and stopped-only disk mutations. Lima only mints it for confirmedStopped; absent, broken, unknown, and failed probes stay distinct. A machine-resource update consumes the stopped token because Lima validates it by booting the VM and Firecracker may boot when--startis requested; the returned operation guard keeps the lock without claiming the VM is still stopped.FirecrackerVm<Configured>/FirecrackerVm<Running>(vm.rs) gatestart()/stop()transitions at compile time.- Setup preflight rejects explicit inputs unsupported by the selected
backend before config loading at the CLI boundary and repeats that capability
check at the backend boundary.
boot_preflight(cfg)then runscfg.validate()before backend setup or boot work, so no VM starts on an invalid config.
backend::SshTarget keeps backend identity separate from its closed transport
choice. TCP targets carry a validated hostname and port. The Firecracker-vsock
variant carries the trusted per-instance Unix-socket path and guest port, and
renders the same Rust __vsock-proxy helper for SSH, SCP, rsync, forwarding,
and generated SSH configuration. The helper bounds and validates Firecracker's
text CONNECT handshake before relaying raw bytes. New Firecracker instances
persist the configured FirecrackerControlMode; the opt-in vsock mode is
then used for readiness, shutdown, and every shared SSH consumer. Older state
without the tag maps to LegacyTcp in memory without an eager rewrite. Lima
remains TCP-based.
Firecracker instances also persist an open or isolated network mode. The
isolated representation uses the composite vsock_isolated value in the
existing control field so supported prior releases reject it instead of
ignoring a new field and booting with a NIC. In memory it normalizes to vsock
control plus an isolated network policy. Firecracker config generation emits an
empty network-interfaces list and backend startup skips TAP setup; every
shared SSH consumer continues through the same vsock target.
An instance operation lock is held from the stopped-state probe through disk resize, commit, or restore. Start, stop, and destroy acquire the same bounded lock. Its file is a sibling of the instance directory, so directory removal does not replace the lock inode. Allocation releases its directory lock before any instance operation lock is acquired.
lib.rs:run() is the entrypoint. Order matters:
- Emit dynamic shell completions (before arg parsing) if requested.
- Parse
Cli(clap derive). - Reject backend-unsupported explicit
setupinputs before tracing, config loading, or other setup side effects. - Init tracing (→ stderr) and handle commands that must work without a
loaded config —
Completions,Init,Update,Uninstall— first. config::CoopConfig::load; on Linux, preflight Firecracker setup tools before managed-storage repair. Non-setup commands prepare storage here. Print the update notice; run its background check except during Linux setup.- Construct
backend::PlatformBackend::new(). matcheachCommandsvariant to acommands::cmd_*handler. Lifecycle handlers callcfg.validate_and_warn()?before VM work; query commands (list/status/logs) skip it.
The commands/ submodules own the domains: lifecycle.rs (up/start/shell/
exec/stop/destroy/status/list/resize/commit/restore), quickstart.rs,
profiles.rs (+ images), agent.rs (coop agent update),
model.rs (coop model), proxy.rs (coop proxy), github.rs,
admin.rs (init/validate/uninstall), and json.rs (machine-readable --json output types). commands/mod.rs
re-exports the dispatch surface and holds cross-domain helpers
(merge_runtime_guest_env, purge_all_data).
The lifecycle is setup → up/start → shell → stop → destroy. A first boot
(lifecycle.rs:start_instance, BootMode::FirstBoot) runs roughly:
- Resolve the target repo/workspace and optionally prompt for a GitHub PAT
(
pat_prompt::maybe_prompt). - Ports — merge forward-port specs and fail fast on host-port collisions
(
port_forward::check_host_port_collisions). Stage explicitguest_filessources in private host temporary directories, resolving permitted symlinks and rejecting invalid source trees before boot or disk replacement. - Boot the VM (
be.create_and_start), thenwait_until_ready(SSH probe with backoff). - Forwards + state — spawn
ssh -Lforwards, persistForwardsState,GuestEnvStateas JSON sidecars in the instance dir. - Guest files and bootstrap —
guest_files.rsuploads the staged copies;scripts/guest/copy-files.shrejects destination symlinks and live-mount overlaps, then merges private, writable files without deletion. Host staging drops at the end of the operation; guest staging is cleaned after each copy. Then (backend.rs:bootstrap_agents): if aGITHUB_TOKENis present,gh auth setup-git; thenbootstrap_claude/bootstrap_codexinject the allowlisted config files and a managedsettings.json, and on first boot install the delta of marketplaces/plugins/MCP servers not already baked into the golden image. - Workspace —
--workspacecopies via rsync;--git-repoclones inside the guest; mounts are live on Lima and rsync'd on Firecracker. PersistWorkspaceState. - Creation hooks — select a recipe before first boot and persist it in
owner-only
creation.json. After workspace provisioning, run globalpost_createin the guest home. Save successful completion atomically. Failures retain the VM and block agent launch; runningupand stopped-instance restarts retry the pending hook under the provisioning lock. The instance operation lock covers the initial state read and the final liveness probe, recipe revalidation, and completion write; it is released while the guest hook runs so stop and destroy can proceed. A hook that succeeds but loses the VM or cannot save completion stays pending and may run again on retry. Guest execution uses a private staged script, a PTY-bound supervisor, and a guest lock; cancellation terminates and reaps the supervised process group. Shell and exec remain available for recovery. post_starthook — runs after creation, workspace, and mount provisioning, with an SSH session prepared after bootstrap to include newly created proxy capability tokens. Hook command failures are warned, not fatal.
Config, secrets, and workspace all cross the host→guest boundary here; the
security-relevant details of each crossing are in
trust-model.md.
config::CoopConfig is the type-safe core. Loading reads TOML (or JSON by
extension) or defaults, then expands user paths. Value bounds are enforced by
newtype constructors (VmMemory has a 128-MiB floor, InstanceIndex is
0..=252, SubnetMask is 0..=32, Hostname/SshUser/EnvVarName validate
their character classes), so validate() only checks environmental facts
(paths exist, binaries are present). Secrets are stored indirected as cmd:
retrieval commands and resolved at VM-start by resolve_cmd_value.
Per-instance runtime state is a set of JSON sidecar files under the instance
dir: instance.json, vm_config.json, workspace.json, forwards.json,
guest_env.json, model.json, proxy.json,
creation.json (selected creation recipe, completion, and deferred
startup hook; mode 0600), plus
the Firecracker .pid/.socket/.log/vsock files.
Allocation refuses an occupied instance path. If any instance directory has
unreadable metadata, allocation stops because its network index cannot be
trusted; existing healthy instances remain available through normal lookup.
update.rs self-updates the CLI and its proxy companion: fetch release metadata
from the fixed https://api.github.com/repos/trailofbits/coop/releases source,
validate the canonical release tag and exact GitHub identity of the platform
tarball, SHA256SUMS, and optional attestations.jsonl before download, verify the checksum (mandatory), verify the Sigstore
attestation via gh against that bundle, fetching a bundle anonymously from
the attestations API by digest when the release publishes no usable one. The
installer uses jq for API extraction and retains the authenticated API path
when jq is absent. Both clients retain authenticated verification if the
anonymous API transfer fails. Verification is skipped only when gh is absent or the
explicit test policy disables it. Both verification paths pin the
signer to .github/workflows/release.yml at the selected tag, require that
source ref, and reject attestations generated on self-hosted runners. Then
extract with path-escape-safe tar flags, and atomically rename the new
coop-proxy beside the CLI before replacing coop. Each replacement is
atomic; the pair is not a single transaction. Official builds cannot replace
the metadata origin at runtime. A separately compiled test policy permits only
a loopback fixture for the host-only updater integration test. A background notifier checks for
new versions on a 24-hour interval (disabled in dev/CI/non-TTY). The full verification chain and
its trust properties are documented in trust-model.md.
The guest runs from a golden image built once and reused (reflink-cloned per
instance on Firecracker). setup.rs builds it via a chroot, running the
embedded provisioning scripts in scripts/guest/ (preamble.sh,
guest-config.sh, cleanup.sh) plus the profile/package installers referenced
from guest.rs. Firecracker uses a minimal CI kernel that is missing several
netfilter modules; the resulting workarounds (iptables-legacy, static
resolv.conf, DOCKER_INSECURE_NO_IPTABLES_RAW=1) are applied in
guest-config.sh and explained in platform-notes.md and
trust-model.md.
These invariants describe the current architecture:
- The VM is the isolation boundary. The guest is untrusted from the host's
view; guest-authored data must not escalate into host code execution or
filesystem escape. See
trust-model.md. - Backend selection is compile-time.
PlatformBackendresolves to Firecracker or Lima at compile time; shared code supports both. - State-gated operations use probes and locks. VM operations use
RunningInstance/StoppedInstanceandboot_preflight; the stopped token holds the operation lock until the disk mutation is complete. - Value invariants live in constructors. Boundaries parse values into newtypes, so downstream code receives values with established invariants.
- Secrets use bounded transport and redacted diagnostics. Env/
SendEnv/stdin carry most secrets; traces redact them. The macOSsecurityand 1Passwordopstore commands are documented argv exceptions (seetrust-model.md). - Remote commands use
RemoteCommand::arg; host commands useCmd::arg. Tainted input is not interpolated into shell command strings. - State is transparent JSON sidecars written atomically
(
fs_util::atomic_write_*) without relaxing permissions.
The Linux network control plane adds a narrower host-launch invariant on top
of item 6. Its sudo, ip, bridge, iptables, and sysctl identities are
resolved from fixed absolute system candidates, validated before use, and
launched from / with an environment cleared to deterministic locale values.
The invocation-selected config may replace any identity with one exact absolute
path for non-FHS layouts. An override never falls back; it retains the full
root-owned, non-writable path-chain checks but intentionally does not claim
package, hash, basename, or command-semantic identity.
setup_tap resolves the complete set before its first probe or mutation.
Firecracker setup also uses the resolver for its fixed system-tool family:
bootstrap tools are checked before storage changes or package installation,
and the installed family is checked afterward, before artifact downloads and
image construction. The privileged disk helper repeats filesystem-tool
resolution at its elevated launch boundary. Other host subprocess families
retain their own launch contracts.