Take the image from 15 MB to under 6

[BryanFRD]

The published image is 15.4 MB compressed (measured on ghcr.io/ferrlabs/lfsx:latest): roughly 10 MB of distroless/cc-debian12 and 5.5 MB of binary. The release artifacts agree, lfsx-server-x86_64-unknown-linux-gnu.tar.gz is 5.21 MB and the musl one 5.41 MB.

Two independent steps get that to somewhere around 5 to 6 MB. They are worth doing in this order, because the first is measurable without touching the runtime and the second is the one that can cost throughput.

Step one: a release profile

There is no [profile.release] section in the workspace at all, so every release binary is built with Cargo defaults: no LTO, codegen-units = 16, symbols kept.

[profile.release]
lto = "fat"
codegen-units = 1
strip = "symbols"

Fifteen to thirty percent off the binary is the usual result, and LTO across crate boundaries tends to be slightly faster rather than slower. The cost is build time in CI, which matters here because a release compiles seven targets.

Not panic = "abort". Today a panic in a handler kills that task and the server keeps serving; with abort it takes the process down. A few hundred kilobytes do not buy that.

Step two: musl on distroless/static

The Dockerfile builds x86_64-unknown-linux-gnu and aarch64-unknown-linux-gnu and lands them on distroless/cc-debian12, which is where the ~10 MB goes. Building the musl targets instead and using distroless/static removes glibc and libgcc entirely, for well under a megabyte of base.

Nothing needs glibc at runtime: rustls is pinned to ring, the trust roots are bundled through webpki-roots rather than read from the system, and zstd is compiled in. The release workflow already produces musl binaries for both architectures, so the toolchain path is proven and only the image would change.

Two things to check rather than assume:

• The musl allocator is slow under heavy multithreaded allocation, which is exactly the shape of a streaming server moving four-megabyte frames through zstd. bench/throughput.sh and the bench.yml workflow already measure this, so the acceptance condition is a run before and after on the same runner. If it moves, mimalloc as the global allocator is the usual answer, and that is a decision worth taking on its own evidence rather than smuggling in with an image change.
• DNS. musl resolves through /etc/resolv.conf and does not do NSS. On Kubernetes that is what happens anyway, but the forge lookups depend on it, so it deserves a real check rather than an assumption.

What this is and is not worth

Operationally, close to nothing: eight megabytes pulled once per node, for a server whose job is to hold gigabytes. What it does buy is the claim in the README being true with a number attached, a smaller runtime surface, and parity with the sub-10 MB figure the nearest alternative advertises.

It ranks below a comparative benchmark harness, and it is small, self-contained and entirely measurable, which is why it is worth writing down rather than doing by feel.

[BryanFRD]

Reopened: the musl half was reverted in #331 because it cannot link in the builder this project actually uses.

zig exhausts file descriptors during its own sub-compilations, and the builder's limit is not raisable from inside:

descriptors: 1024 soft, 1024 hard
error: sub-compilation of compiler_rt failed
  note: unable to load source: ProcessFdQuotaExceeded

Docker Desktop hands out 1048576, which is why every local build of the same Dockerfile succeeded, on both architectures, image measured at 4.4 MB against the 12.8 MB the glibc one costs. The size win is real and reachable; the method was not.

Two candidates for the next attempt, neither of them a Dockerfile change:

• Raise nofile where the build container is created. That is --ulimit on the buildah invocation inside FerrLabs/.github's reusable image workflow, so it needs a change there and would help any other project that meets this.
• Drop zig for a musl cross toolchain that does not spawn a compiler per runtime library. Debian packages none for aarch64, so this means pinning a third-party toolchain and weighing that against the eight megabytes.

The acceptance condition changes too: a green build in this CI, not on a laptop. The first step of the issue (#328, the release profile) is merged and unaffected.

[BryanFRD]

Done, and measured on the registry rather than a laptop.

ghcr.io/ferrlabs/lfsx:1.16.6 is the first musl image, published by the v1.16.6 release and gated by the smoke test before it was pushed:

	amd64	arm64
1.16.5, glibc on cc-debian12	13.5 MB	13.3 MB
1.16.6, musl on static-debian12	4.5 MB	4.2 MB

Both steps are in. Step one, the release profile, was #328. Step two took two attempts: #329 built on a laptop and died in CI, #331 reverted it, and #347 landed it once FerrLabs/.github#327 added a build-ulimits input to the reusable image workflow. The blocker was never the toolchain, it was that buildah gives a RUN 1024 descriptors, hard, and cargo-zigbuild holds one open per source it compiles for compiler_rt. A hard limit cannot be raised from inside the process it constrains, which is why the Dockerfile could not fix it and Docker Desktop, handing out 1048576, never showed the problem.

The two things this issue said to check rather than assume:

DNS and TLS work from distroless/static. With LFSX_AUTH=github and the default API root, a batch request with a junk token comes back 401: the server resolved api.github.com, handshook against the compiled-in webpki roots, and GitHub rejected the token. Pointed at a host that cannot resolve, the same request comes back 502 with forge request failed. Two different outcomes, so the 401 is a real round trip.

The musl allocator is a question this hardware cannot answer. The host and musl legs of bench.yml both pass, but four runs of the same commit spread upload from 150 to 273 MiB/s and small objects from 1.3 to 16 ms. Nothing at this scale is resolvable above that noise, so there is no evidence of a regression and equally none that would justify mimalloc. docs/performance.md says so rather than quoting a number the runners did not earn.

Closing this. #350 carries the follow-ups from review: the zig tarball checksum, the toolchain layers moved above the COPY lines so a release stops recompiling them, a pull-request job that runs the image rather than only building it, and the README claim with the number above attached.