Status: P1 to P5 built. The generic urunc layer (P1-P4) was verified on xob0 (Ubuntu 24.04, amd64) and rpi5-6 (Debian 12, Raspberry Pi 5, arm64); P5 turned it into the self-contained brig installer that is now the shipped product. See README.md for the shipped behavior, "Decisions taken during implementation" for where this document was overtaken, and "P5: the brig installer" below for the conversion. Target issue: NOFireAI/engineering#1272 (the brig self-contained bundle) Scope of this document: the generic urunc layer and its conversion to brig.
The generic layer was always meant to be re-skinned for brig (P5 below). That is
now done and is what install.sh and scripts/build-bundle.sh produce.
- One root, laid out like k3s. Everything lives under
/var/lib/brig, as two sibling trees:data/holds the immutable binaries, config and boot assets (the unpacked tarball) andagent/holds the containerd store, snapshots and pool. The socket stays in/run/brigand the launchers on the hostPATH, exactly as k3s keeps its socket in/run/k3sand its binary in/usr/local/binwhile everything else sits under/var/lib/rancher/k3s. This replaces the earlier/opt/brig+/var/lib/brigsplit: the immutable/mutable separation is kept, but asdata/vsagent/under one root rather than two top-level directories. - One tarball, built by the release, unpacked by the installer. The unit of
distribution is a single per-arch tarball published by this repository's release
workflow. It is a complete
/var/lib/brig/datatree: every binary, every config file, the systemd units, the wrappers and the boot assets, laid out and ready.scripts/build-bundle.shis where all the assembly and building happens;install.shfetches that one tarball, verifies it, unpacks it, and wires it into the host. The installer builds nothing, runs no compiler and no docker, and fetches nothing else. Because the config files carry the fixed layout, the paths above are not configurable at install time. - Why urunc is built (at release). Most components are upstream release
artifacts, hvi among them (
brig-sh/hvi-vmmv0.1.0-rc.2, cosign-verified). urunc has no release that carries what brig needs, so the build compiles it fromurunc-dev/uruncat commit3c00b69, onfeat/linux-hvi-v050(CGO static, so native per arch under qemu/binfmt). The source is pinned to a commit, since a branch tip moves with every push. The README's build section says how to move the pin. The kernel and the initrd are fetched fromghcr.io/nofireai/hull-assetson both arches, by digest, once its cosign signature checks out. The initrd is urunit alone, and its agent has to match the urunc shim. hull-assets records the urunc it built from in its signedbundle.json, and the build refuses a bundle whose urunc is not the one in the tarball. - The installer's own job is the host wiring the tarball cannot carry: create
the state and socket dirs, provision the device-mapper pool when the kernel
supports it (brig defaults to overlayfs;
CONTAINERD_SNAPSHOTTER=devmapperswitches at runtime), create thebriggroup, drop the/etc/urunc/config.tomlsymlink and the/usr/local/binlaunchers, write the.install-stamp, and -- after asking -- enable the systemd service and start it. It runs quiet with a progress bar by default, or verbose with a line per stage. - Non-root via a group. A
briggroup owns the private containerd socket: the systemd unit chgrps the socket to that group by name once containerd is up (no gid is baked intocontainerd.toml, so the tarball stays host-independent). A member of the group runsbrigthrough the/usr/local/bin/briglauncher, which sources/var/lib/brig/data/etc/brig-env.sh(pointing nerdctl at the private containerd and brig at the packed boot assets) and execs the real binary.
The rest of this document describes the generic urunc layer it is built on.
Issue #1272 describes the brig tree: a private containerd, the brig-urunc
shim, nerdctl, static VMMs, everything under /var/lib/brig,
nothing shared with the host. That is the right end state, but most of the work
in it is not brig-specific. It is "package urunc and everything it needs, then
put it on a machine in one command".
So we should build the generic thing first. One arch-specific tarball with
stock urunc and its dependencies, a k3s-style curl | sh installer, a
matching uninstaller, and devmapper set up for us rather than by hand. Once
that works, brig re-skins it: different root dir, different runtime class
name, a couple of extra binaries.
Installing urunc today means following
docs/installation.md
by hand. That document is ~500 lines and it touches the host's containerd
config, the host's /usr/local/bin and the host's thinpool. It works, and
every step of it is a step we can automate.
Four things changed once this went from design to a working installer.
One base directory, not three. Everything lives under /var/lib/urunc:
binaries, configuration, containerd state, the socket and the pool backing
files. The three-way split proposed below is the k3s shape, but a single root
is what makes uninstall a single rm -rf plus three recorded host objects.
No LVM pool. Loop backed files only. The lvm mode described below was
dropped.
Sparse backing files. truncate, not fallocate, so a 100G pool costs
nothing until it is used. The measured thin-pool stall is real, so
INSTALL_URUNC_POOL_PREALLOC=true switches back to fallocate for anyone who
hits it.
No QEMU. Not installed. INSTALL_URUNC_MONITORS can add it back from the
same monitors release. virtiofsd is kept, at 3MB, because it is the only rootfs
path left when the host has no device-mapper thin pool.
/etc/urunc/config.toml is left as a symlink into the tree, as the design
proposed. The upstream URUNC_CONFIG patch is still worth sending, and is not
needed for this to work.
Four things that only showed up on a real host. All four are folded into
install.sh.
-
Pulls fail without
unpack_config. A hand written containerd config has no[[plugins.'io.containerd.transfer.v1.local'.unpack_config]]entry, and every pull dies withno unpack platforms defined. The installer writes one for the chosen snapshotter and one for overlayfs. -
nerdctl's default bridge collides with the host. nerdctl creates its default network on 10.4.0.0/24, which a host nerdctl has usually already taken. The installer ships its own conflist on 10.44.0.0/24 with a
urunc0bridge, both configurable. -
The snapshot filesystem has to be ext2. Rumprun guests support nothing else (
pkg/unikontainers/unikernels/rumprun.go), and with ext4 urunc fails with "can not use the container rootfs as the sandbox's guest rootfs through block or shared-fs". Linux guests read ext2 as well, so ext2 is the one default that works for every guest. -
solo5-hvt dies under the default seccomp profile on Ubuntu 24.04. Exit 159, SIGSYS. urunc applies its own hardcoded allowlist before executing the tender, with
DefaultAction: ActionTrap, and upstream documents that allowlist as tested on Ubuntu 20.04 and 22.04 only. Both solo5 v0.9.3 and v0.12.1 fail identically here, and both boot with the profile off, so this is neither the packaging nor the solo5 version. It is the one thing the installer does not fix, and the README says so.
Surveyed against urunc-dev/urunc main at 609e07e. Four groups:
| Group | Components | Where they come from |
|---|---|---|
| urunc itself | urunc, containerd-shim-urunc-v2 |
urunc releases (static, per-arch) |
| monitors | firecracker, cloud-hypervisor, solo5-hvt, solo5-spt, qemu + its firmware blobs, virtiofsd | urunc-dev/monitors-build releases |
| container plumbing | containerd, ctr, runc, nerdctl, CNI plugins | upstream releases |
| host bits we do not ship | /dev/kvm, device-mapper, iptables, a kernel with tun/tap |
the distro |
Two things fall out of this that make the job much smaller than it looks.
First, upstream already publishes both halves of the artifact set. urunc
releases carry urunc_static_$ARCH and containerd-shim-urunc-v2_static_$ARCH
(12-14MB each). monitors-build publishes a single per-arch tarball with all
five monitors plus virtiofsd, already static, already versioned in the tag
name, e.g. FC-v1.7.0_CLH-v50.0_S5-v0.12.1_VFS_-v1.13.0_QM-v10.1.1-9a44e at
42MB compressed for amd64. So this is an assembly job, not a build farm. We
pin, fetch, verify and lay out. Building from source stays available behind a
flag for the cases where we need a patched urunc.
Second, runc is only needed for the Kubernetes path, where urunc delegates
pause and sidecar containers to it (cmd/urunc/utils.go:98, an exec.LookPath).
For a standalone non-k8s install it is still worth shipping, since nerdctl
users will want normal containers next to unikernels.
These came out of reading the tree and poking the test hosts. Each one changes a decision.
1. /etc/urunc/config.toml is a hardcoded constant.
// pkg/unikontainers/urunc_config.go:25
const UruncConfigPath = "/etc/urunc/config.toml"There is no env var and no flag. Both call sites (cmd/urunc/create.go:77 and
pkg/containerd-shim/guest_rootfs.go:62) use the constant directly. So urunc
is relocatable in every path except its own config, and there can only be one
urunc configuration per host.
For a stock bundle the workaround is a symlink: the real file lives at
$URUNC_ROOT/etc/config.toml and /etc/urunc/config.toml points at it.
toml.DecodeFile opens through symlinks, so this works with no patch. The
installer records whether it created that symlink, and the uninstaller removes
it only when it still points into our tree.
We should also send a small patch upstream that honors a URUNC_CONFIG
environment variable and falls back to the constant. That is the one change
that makes two urunc installs coexist on a host, and brig will need it.
2. containerd, nerdctl, runc and the CNI plugins are fully relocatable.
containerd takes root, state and grpc.address in its config. nerdctl
reads $NERDCTL_TOML and honors $CONTAINERD_ADDRESS, $CONTAINERD_NAMESPACE,
$CONTAINERD_SNAPSHOTTER and $CNI_PATH. Verified on xob0 with the installed
nerdctl 2.0.3 and containerd 2.2.6. So a private stack needs no patched
binaries, only a config file and a systemd unit with the right Environment=
lines.
containerd resolves the shim by looking up containerd-shim-urunc-v2 on its
own PATH. That is the hook we use: the private containerd unit gets
Environment=PATH=$URUNC_ROOT/bin:/usr/sbin:/usr/bin:/sbin:/bin, and the
private stack picks up the bundled shim while the host containerd keeps
picking up whatever it had.
3. All three test hosts already run a host-wide urunc and a live thinpool.
nofire, xob0 and nofire-cerno are all Ubuntu 24.04 / x86_64 with
containerd 2.2.6. Each has /usr/local/bin/urunc, /etc/urunc/config.toml,
monitors in /opt/urunc/bin, and a devmapper pool in use: devpool on xob0
(loop-backed, 200GB), ubuntu--vg-devpool and a live k3s-devpool on the
other two.
So coexistence is a hard requirement, not a nice-to-have. The installer must
not write the host's /etc/containerd/config.toml, must not touch the host
pool, and must not restart the host containerd. Uninstall must leave all of
that exactly as it found it. These hosts are a good test bed precisely because
they will notice if we get it wrong.
It also means the default root cannot be assumed empty. /opt/urunc is
already populated on all three.
4. The stock thinpool script is the slow configuration.
script/dm_create.sh creates the backing files with truncate, which leaves
them sparse, on whatever filesystem /var/lib/containerd sits on, then loops
them. On ext4 that puts thin-pool metadata writes behind jbd2 commits and
create_snap stalls for tens of seconds. Measured previously: fallocate
instead of truncate cuts the stall rate from ~100% to ~16%, and moving off
the loop file onto a real block device removes it.
So we should not ship dm_create.sh as-is. The installer prefers a real
device, falls back to fallocated loop files, and never uses truncate.
5. QEMU's firmware blobs are most of the bundle.
On xob0 the installed monitor set is 76MB of binaries, 3.2MB of virtiofsd and
273MB of QEMU share/. So a full bundle is ~380MB installed, and a bundle
without QEMU is ~90MB. That argues for two flavors, with QEMU opt-in. It also
matches #1272, which already wants QEMU treated as an optional plugin path.
/var/run/urunc cannot hold the tree. /var/run is a symlink to /run,
which is tmpfs, so it is cleared on every boot. Sockets belong there, 380MB of
firmware blobs and an image store do not. We should use the three-way split
that k3s uses: immutable artifacts in one place, mutable state in another,
runtime sockets in tmpfs.
/opt/urunc/ $URUNC_ROOT -- immutable, replaced wholesale on upgrade
bin/ urunc, containerd-shim-urunc-v2, containerd, ctr,
nerdctl, runc, firecracker, cloud-hypervisor,
solo5-hvt, solo5-spt, qemu-system-$ARCH
libexec/ virtiofsd
libexec/cni/ CNI plugins
share/qemu/ QEMU firmware blobs
etc/config.toml urunc config (the symlink target)
etc/containerd.toml private containerd config
etc/nerdctl.toml private nerdctl config
etc/cni/net.d/ private CNI network
pins.env every bundled version, one KEY=value per line
.install-stamp what this install created, for the uninstaller
/var/lib/urunc/ mutable state
containerd/ containerd root (images, snapshots, content store)
devmapper/ thinpool backing files, loop mode only
/run/urunc/ ephemeral
containerd/ containerd state dir
containerd.sock the private socket
/etc/urunc/config.toml symlink -> /opt/urunc/etc/config.toml
/usr/local/bin/urunc-ctl wrapper: exports the env and execs ctr/nerdctl
/usr/local/bin/urunc-uninstall.sh generated at install time
Everything except /etc/urunc/config.toml moves with $URUNC_ROOT, so a
second install for testing goes to /opt/urunc-test and collides with nothing
but the symlink.
pins.env is the manifest #1272 asks for, and the single one this install
carries: it is what brig-ctl version prints. (An earlier design also emitted a
components.json and per-bundle SBOMs; both were dropped as redundant with
pins.env.)
The upstream urunc-deploy convention already uses /opt/urunc/bin and
/opt/urunc/share for monitors, and all three test hosts have that layout.
Our tree is a superset of it, so the paths do not conflict in shape, but a
hand-made install has no .install-stamp.
The installer should therefore refuse to write into an existing $URUNC_ROOT
that it did not create, and print what it found and how to proceed. Two escape
hatches: INSTALL_URUNC_FORCE=1 to take it over, or a different
INSTALL_URUNC_ROOT. Silent adoption is how we would corrupt somebody's
working setup.
curl -sfL https://get.urunc.io/install.sh | sh -
Knobs, k3s naming so it reads familiar:
| Variable | Default | Meaning |
|---|---|---|
INSTALL_URUNC_VERSION |
latest release | urunc version to install |
INSTALL_URUNC_CHANNEL |
stable |
stable or nightly |
INSTALL_URUNC_MONITORS_VERSION |
pinned tag | monitors-build release tag |
INSTALL_URUNC_ROOT |
/opt/urunc |
bundle root |
INSTALL_URUNC_DATA_DIR |
/var/lib/urunc |
mutable state |
INSTALL_URUNC_FLAVOR |
full |
full or slim (slim drops QEMU, ~90MB) |
INSTALL_URUNC_SNAPSHOTTER |
devmapper |
devmapper, blockfile or overlayfs |
INSTALL_URUNC_POOL_MODE |
auto |
lvm, loop, existing or none |
INSTALL_URUNC_POOL_DEVICE |
unset | block device or VG to carve the pool from |
INSTALL_URUNC_POOL_SIZE |
100G |
pool data size |
INSTALL_URUNC_TARBALL |
unset | install from a local tarball, no network |
INSTALL_URUNC_SKIP_START |
false |
lay the tree down, do not start containerd |
INSTALL_URUNC_FORCE |
false |
take over a foreign $URUNC_ROOT |
What it does, in order:
- Preflight. Check root, arch (amd64/arm64),
/dev/kvm, thedm_thin_poolmodule,iptables, and that$URUNC_ROOTis either absent or ours. Fail with one clear line each, before touching anything. - Resolve versions and write
pins.env. - Fetch the urunc binaries, the monitors tarball, containerd, runc, nerdctl and the CNI plugins into a temp dir. Verify checksums, and signatures where upstream publishes them.
- Lay out
$URUNC_ROOTatomically: unpack into$URUNC_ROOT.new, then rename. A re-run is an upgrade, and a failed re-run leaves the old tree. - Set up storage (below).
- Write the four config files, then the
/etc/urunc/config.tomlsymlink. - Install
urunc-containerd.service, plus the pool reload unit in loop mode. - Generate
urunc-uninstall.shfrom the stamp. - Start, then verify: the private
ctr plugin lsshows the snapshotterok, and a unikernel image runs end to end.
Offline install matters for the airgapped hosts, so INSTALL_URUNC_TARBALL
takes a pre-staged bundle and skips every download.
Three modes, chosen by INSTALL_URUNC_POOL_MODE, defaulting to auto.
lvm, preferred. Given a VG with free extents, or a whole device in
INSTALL_URUNC_POOL_DEVICE, create a thin pool LV. No loop devices, no jbd2 in
the path, and LVM reactivates it at boot, so no reload unit is needed. This is
what auto picks when it finds free extents.
loop, fallback. fallocate (never truncate) the data and metadata
files under $DATA_DIR/devmapper, losetup them, dmsetup create the pool.
Install urunc-devpool.service to re-create the pool before the private
containerd starts, ordered Before=urunc-containerd.service. This is stock
dm_create.sh with the sparse-file bug fixed and the naming made unique.
existing, for a pool the operator already manages. We only write the
containerd config, and the uninstaller never touches the pool.
none falls back to overlayfs. urunc still runs, but the block-rootfs
paths are unavailable, so we print that plainly.
The pool is named from the root: urunc-devpool by default. It must not be
containerd-pool or devpool, both of which are already taken on the test
hosts.
One extra check the stock scripts do not do. A leaked dm-thin metadata
snapshot poisons every later create_snap on that pool, and it presents as
unrelated container failures. Field 7 of dmsetup status <pool> is the tell:
- when clean, a block number when one is held. The installer checks it after
creating the pool, and the uninstaller releases one before removing the pool.
urunc-uninstall.sh, generated at install time, driven by .install-stamp. It
removes what we created and nothing else. Order matters, because a pool with
live thin devices will not go away:
systemctl stop urunc-containerd.service, disable both units, remove them.- Delete containers, tasks and snapshots in the private containerd namespaces.
- Release a held metadata snapshot if
dmsetup statusshows one. dmsetup removeeveryurunc-devpool-snap-*, then the pool itself.losetup -dour loop devices, orlvremoveour LV. Nothing else.- Remove
/etc/urunc/config.tomlonly if it still points into our tree. - Remove
$URUNC_ROOT,$DATA_DIR,/run/uruncand the wrappers.
--keep-data keeps $DATA_DIR for a reinstall. The script is idempotent, and
a second run on a half-removed install finishes the job rather than failing.
xob0 is the primary test host: 753GB free, no k3s, and a host urunc install to
prove we do not disturb. The others are for the k3s-adjacent case, where the
host runs a live k3s-devpool that we must leave alone.
Install to /opt/urunc-test with INSTALL_URUNC_DATA_DIR=/var/lib/urunc-test,
so the host's own /opt/urunc is out of the picture for the first pass.
The two pool modes fall out of the hosts for free. xob0 has no LVM volume
groups at all, so auto picks loop there. nofire and nofire-cerno both have
ubuntu-vg, so auto picks lvm. That covers both paths without contriving
anything.
Positive tests:
- the private
ctr -a /run/urunc/containerd.sock plugin lsshows devmapperok - a unikernel image runs to completion through the private stack, on each of firecracker, cloud-hypervisor, hvt and spt
- a normal OCI container runs through the same stack via nerdctl, with working network
- re-running the installer upgrades in place and does not lose the image store
Negative tests, which are the ones that actually matter here:
- host
/etc/containerd/config.tomlbyte-identical before and after - host containerd never restarted,
systemctl show -p ActiveEnterTimestampunchanged - host
devpoolstill listed and still healthy after our install and after our uninstall - installer refuses a foreign
$URUNC_ROOTwithoutINSTALL_URUNC_FORCE - uninstall on a host where the pool was
existingleaves the pool alone - after uninstall,
losetup -aanddmsetup lsshow exactly what they showed before install
- P0 -- design. This document. Decisions in the open questions below.
- P1 -- bundle and install. Fetch, verify, lay out, private containerd,
nerdctl, configs, service. Snapshotter
overlayfsonly, so the storage work does not block the first end-to-end run. - P2 -- storage. The three pool modes, the reload unit, the metadata-snap check.
- P3 -- uninstall. The stamp, the generated script, the negative tests.
- P4 -- release. A workflow that builds the per-arch tarballs, publishes
install.sh, recordspins.env, and cosign-signs the checksums. - P5 -- hand to brig. Parameterize root, runtime class and socket path, and close the generic half of #1272.
P1 through P4 are built. P1 to P3 are verified end to end on xob0; P4's bundle build, reproducibility, offline install and signature refusal are verified there too, while the release and CI workflows themselves are unrun until this lands in a repo. P5 is open.
- Where does this live? A new repo, or in-tree in
urunc-dev/uruncunderdeployment/standalone/? In-tree is where it belongs long term, since it is stock urunc and upstream already carriesurunc-deploy. Starting in a NOFireAI repo is faster and lets us move without upstream review. I would start in-tree on a branch and see how the first review goes. - Bundle containerd, or attach to the host's? I would bundle. It is what #1272 wants, it is the only way uninstall is clean, and the test hosts prove the coexistence case is real. Attaching to an existing containerd can be a second mode later.
- QEMU in the default flavor? It is 273MB of the 380MB. I would default to
slimand makefullexplicit, given #1272 already treats QEMU as optional. - Do we send the
URUNC_CONFIGpatch upstream now? It is a ten-line change and it removes the only non-relocatable path in urunc. I think yes, early, so the symlink stays a fallback rather than the design. - Where does
install.shget served from?get.urunc.iowould match the k3s feel. A raw GitHub URL works on day one.