You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use containerd's content store as Racer's disk copy for Gantry - #1111
This replaces the standalone cachemesh routing library from #1106 by building the same idea into Racer, as suggested in review. A new ClusterCache.spec.persistence: Adapter mode keeps Racer pages in memory only and asks the local origin adapter for a page first (a pinned origin GET with Racer-Source: local), so for Gantry the containerd content store becomes the only on-disk copy; page owners also remember which nodes recently fetched a page and ask those holders before going back to the registry. Racer stays OCI-agnostic and gains a client Invalidate call that drops one cached version across its page owners.
On the Gantry side, Racer mode now checks the caller's credentials against the upstream registry before serving cached content (fixing a gap where any non-empty token could read cached private blobs), and serves local-only page requests from a read-only mount of containerd's content store after verifying each blob's sha256 once. Gantry also verifies the sha256 of manifests and full blob responses it serves from Racer, aborting and invalidating the Racer version on a mismatch. The PR also adds a cluster benchmark harness for comparing default, Racer, and Adapter modes, plus designs/racer-containerd-store.md and updated Racer and Gantry docs.
Racer serves cache hits by digest with no per-request credential check.
Before consulting Racer, HEAD the exact repository and digest upstream with
the caller's delegated Authorization (or none) and cache the decision briefly
per credential hash, registry, repository, kind, and digest.
Racer treats a version's content type as immutable and compares it across
pages. Pages read from the local containerd content store cannot know the
registry's Content-Type header, so registry pages now report blobs as
application/octet-stream too, keeping both sources on one version.
Persistence Adapter keeps a cache's pages in Racer memory only and relies on
the origin adapter's local copies. The CRD rejects it together with
passthrough or origin; the authority maps it to the new CacheMode::Adapter
(wire spelling "adapter"), and publications reject it with a native origin.
A pinned page GET with Racer-Source: local asks the origin adapter to serve
the page only from a copy it holds locally, never from its remote source. A
well-framed 404 or 412 is a local miss. The Go SDK exposes the flag as
OriginRequest.LocalOnly and answers a local-only ErrNotFound or
ErrVersionMismatch with 404.
CacheMode::disk() is false for adapter caches as for passthrough. The store
refuses their writes and pending copies, fills never persist or refresh
them, disk lookups and cuFile staging skip them, reads do not spill ahead to
disk holds, prefetch admits them to memory only, and each snapshot purges
their disk records. Unlike passthrough, pages and metadata stay in memory.
On a memory miss, Fill::acquire_once now visits racer-core's local_sources
before the ranked owner walk. For persistence Adapter that is the node's own
origin adapter, asked with a local-only page request
(Origin::local_page_reserved). A hit is encrypted and published to memory
like an origin page; a miss or failure falls through to peers and the remote
origin without origin failure notes or link-health accounting. Both readers
and owners filling for peers take this path.
DELETE /v1/objects/<key> with a strong If-Match drops the version's metadata,
memory pages, queued writes, and disk index entries on this node, then sends a
signed prefetch 'invalidate' action to each page's ranked owners. Receivers
drop local state only and never forward it.
An invalidation of a version on a worker now also forgets every recorded
holder of each invalidated page, so a later fill does not probe holders
for a version the client declared stale.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Empty-metadata probes cannot recover evicted blobs from containerd, and incorrect YAML indentation breaks benchmark deployment with a custom Gantry image.
Extends Racer with adapter-backed persistence so Gantry can reuse containerd’s content store instead of keeping a second disk copy, bringing the approach proposed in #1106 into Racer.
Changes:
Adds Adapter persistence, local-only page reads, holder probes, and version invalidation.
Adds upstream credential checks and full-response digest verification for Gantry’s Racer backend.
Adds benchmark tooling, regression tests, and documentation.
File
Description
pkg/racersdk/origin.go
Exposes local-only origin requests and maps local misses.
pkg/racersdk/origin_test.go
Tests local-only request handling.
pkg/racersdk/invalidate.go
Adds client-side version invalidation.
pkg/racersdk/invalidate_test.go
Tests invalidation requests, errors, and cancellation.
pkg/racersdk/internal/wire/wire.go
Encodes and validates local-only headers.
pkg/racersdk/internal/wire/wire_test.go
Tests local-only wire compatibility.
pkg/racersdk/internal/wire/invalidate.go
Implements invalidation request and response encoding.
…fixture reads
The retained-identity startup test gave bootstrap a 30s scope, but seat
403 rejections retry 5 times with ~1+2+4+8+16s backoff, so loaded runs
hit DeadlineExceeded before the retry bound reported Unauthorized. Use a
120s scope. Also raise the portable adapter fixture client socket
timeouts from 2s to 30s.
A holder whose memory copy was evicted asked its local adapter with no
metadata, so an adapter that names objects by metadata (Gantry) could
never serve its durable copy. Owners now forward the reader's opaque
adapter metadata, never authorization, in holder probes, and holders
pass it to their local-only adapter request.
Without metadata the key and ETag cannot distinguish a manifest from a
blob, so no Content-Type is guaranteed to match the registry's. Answer
ErrNotFound so Racer moves on instead of failing the read.
The verifying writer only forwards registry object bytes, which are already served as attachments with nosniff. Holding the destination as an io.Writer keeps CodeQL from treating the passthrough as an HTML response sink, and the stale comment about digest verification now matches the full-object check.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Local replica publication errors and stalls can prevent fallback to healthy sources, while inconsistent manifest Content-Type metadata can cause valid pulls to fail.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Unbounded probe timeout blocks peer and origin fallback
cmd/racer/src/read/fill.rs:2967
A local adapter that accepts the socket but stalls can consume the entire read deadline because this optional probe uses the parent scope for both headers and body. Its DeadlineExceeded is propagated at line 2993, so healthy peers or the remote origin are never tried. Give the probe a bounded child scope and treat that child's timeout as a local failure while the parent remains live.
Publication errors abort peer and origin fallback
cmd/racer/src/read/fill.rs:2985
A local replica whose immutable metadata conflicts with the retained version reaches publish, which returns CorruptRecord; this ? then fails the whole read instead of trying peers or the remote origin. Move publication into attempt so that publication failures use the existing local-failure fallback.
Manifest media type mismatch breaks immutable metadata
internal/gantry/racer/local.go:171
A manifest's JSON mediaType can omit parameters from the registry's Content-Type, such as charset=utf-8, while originContentType preserves that header (racer.go:295-300). Returning the unparameterized value under the same ETag conflicts with Racer's immutable metadata checks and can fail a valid pull. Preserve the authoritative manifest type across both sources, or return a local miss when it cannot be reproduced.
Reduce manifest content types to the bare lowercase media type on both the registry and local content store paths so Racer version metadata agrees for the same ETag.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unresolved issues break custom-namespace setup, allow corrupt adapter replicas to cause repeated verification failures, and reuse incorrect media types across blob and manifest routes.
No open issues 🎉
3 previously missed issues
In code that hasn't changed since last review
Render BENCH_NS consistently in both manifests
hack/gantry-cluster-bench/registry.sh:35
BENCH_NS is configurable, but both applied manifests hard-code gantry-bench, and this rendering only substitutes the node and port. Setting another namespace therefore deploys the registry in gantry-bench while the rollout wait and all subsequent registry commands target BENCH_NS, so setup fails. Render the configured namespace into both manifests (and the namespace creation used by the DaemonSet helpers).
Preserve manifest media type across digest cache hits
internal/gantry/racer/local.go:152
Request keys objects only by digest (racer.go:88), but this descriptor changes with the request kind. After a blob-route read caches a manifest as application/octet-stream, a manifest-route read of the same digest reuses that type (racer.go:858) instead of its manifest/index media type, which can break image resolution. Make cached metadata route-independent and recover the manifest media type at the HTTP boundary, or separate cache identities; add a blob-then-manifest regression test.
Bypass corrupt adapter replicas during recovery reads
internal/gantry/racer/verify.go:152
When a multi-page blob is corrupt in an adapter's containerd store, Invalidate clears Racer state but serveLocal can reopen the same bad bytes on the next read (internal/gantry/racer/local.go:111). This can make retries repeatedly fail digest verification even if registry content is healthy. Add a quarantine or bypass mechanism for suspect adapter replicas, including those on remote owners, so recovery reads can reach the registry.
Hash each content store file once, keyed by file identity, before serving
it for a local-only request. Corrupt files are quarantined as local misses
until replaced, so recovery after Invalidate can reach peers or the
registry instead of rereading bad bytes on this node or a remote owner.
A mirror digest mismatch drops the local result so the file is rechecked.
Adds gantry_racer_local_replica_verifications_total{result}.
Racer keys objects by digest alone, so a digest filled through the blob
route was later served on the manifest route as application/octet-stream.
Store application/octet-stream for every kind from both the registry and
local content store, and recover the manifest media type at the HTTP
boundary from the buffered, digest-checked manifest body (mediaType field,
else inferred OCI index or manifest; untypeable bodies get 503 so
containerd falls through to the registry).
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
The fallback open omits O_NONBLOCK, allowing a FIFO at a digest path to block indefinitely before validation and occupy an origin callback slot despite the request deadline.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The coupled Go/Rust persistence, peer-recovery, and credential-enforcement changes require human review and multi-node validation of mode transitions and corruption recovery under concurrent pulls and peer failures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
This replaces the standalone cachemesh routing library from #1106 by building the same idea into Racer, as suggested in review. A new
ClusterCache.spec.persistence: Adaptermode keeps Racer pages in memory only and asks the local origin adapter for a page first (a pinned origin GET withRacer-Source: local), so for Gantry the containerd content store becomes the only on-disk copy; page owners also remember which nodes recently fetched a page and ask those holders before going back to the registry. Racer stays OCI-agnostic and gains a clientInvalidatecall that drops one cached version across its page owners.On the Gantry side, Racer mode now checks the caller's credentials against the upstream registry before serving cached content (fixing a gap where any non-empty token could read cached private blobs), and serves local-only page requests from a read-only mount of containerd's content store after verifying each blob's sha256 once. Gantry also verifies the sha256 of manifests and full blob responses it serves from Racer, aborting and invalidating the Racer version on a mismatch. The PR also adds a cluster benchmark harness for comparing default, Racer, and Adapter modes, plus
designs/racer-containerd-store.mdand updated Racer and Gantry docs.