Skip to content

Fix gateway registration wait lifetime - #7150

Open
david-piholyuk wants to merge 1 commit into
moby:masterfrom
david-piholyuk:fix/gateway-registration-deadline
Open

david-piholyuk wants to merge 1 commit into
moby:masterfrom
david-piholyuk:fix/gateway-registration-deadline

Conversation

@david-piholyuk

Copy link
Copy Markdown

Problem

The gateway client starts its initial Ping concurrently with Control/Solve. Ping can arrive before Solve registers the gateway bridge. The registrar currently discards that pending entry after three seconds, although the initial Ping has a fifteen-second deadline. A delayed but valid registration can therefore fail as NotFound: forwarding Ping: no such job before the request's deadline.

This addresses the registration-timeout failure mode discussed in #5171. It does not claim to fix every possible cause of that error.

Change

  • Keep pending registrations alive while callers are waiting, subject to their context.
  • Remove an unregistered entry as soon as its last waiter leaves, instead of starting a per-ID timer goroutine.
  • Coordinate Register, Discard, and waiter cleanup under the registrar mutex, so cancellation cannot remove a concurrently published value or affect a replacement entry.
  • Bound gateway lookup to fifteen seconds, matching the existing initial Ping budget. Earlier caller deadlines still apply. An unknown ID without a caller deadline cannot wait indefinitely.
  • Preserve early registration before build-history bookkeeping and update its obsolete timeout comment.

An unknown ID now reaches the lookup/request deadline rather than being discarded with a synthetic cancellation after three seconds. Actual Discard still wakes waiting callers with cancellation.

Reproduction and validation

The new TestDelayedRegistration reproduces the ordering with virtual time: Get begins immediately, Register occurs after four seconds, and the caller allows fifteen seconds. It fails against the old registrar and succeeds with this patch. The abandoned-lookup test also fails against the old registrar.

I additionally ran a local end-to-end Buildx reproduction using an HTTP/2 proxy that forwards Ping immediately but delays only Control/Solve by four seconds. The Dockerfile uses FROM scratch and copies one proof file; no external image pull, registry, daemon restart, or application build is involved.

Daemon No injected delay Four-second Solve delay
Official v0.31.1 Pass Ping fails after about three seconds
Unpatched v0.33.0, built from source Pass Same Ping failure after about three seconds
Patched v0.33.0, built from source Pass Pass; exported proof file matches

For the v0.33.0 comparison, both daemons were compiled from the same release source using Go 1.26.3 and CGO disabled, and run in identical container packaging. The patched end-to-end case passed twice. These are fault-injection correctness tests, not build-performance benchmarks.

Validated again after applying the patch to current upstream master:

go test -race -count=100 ./util/registrar ./control/gateway
go vet ./util/registrar ./control/gateway
git diff --check

Tests cover delayed/missing registration, shorter request deadlines, cancellation cleanup, multiple waiters, discard/replacement, retention of registered values, and concurrent cancellation/registration. The full BuildKit integration matrix and make validate-all have not been run locally.

Wait for a pending gateway registration within the request deadline instead of discarding its ID after three seconds. Bound gateway lookups to the initial Ping budget, and remove abandoned pending entries when their last waiter leaves.

Serialize registration and waiter cleanup to prevent cancellation from deleting a completed registration. Cover delayed registration, deadline and cancellation cleanup, concurrent waiters, discard/replacement, and register/cancel races.

Signed-off-by: David Piholyuk <240580814+david-piholyuk@users.noreply.github.com>
@david-piholyuk
david-piholyuk force-pushed the fix/gateway-registration-deadline branch from aa764c0 to 040554e Compare September 15, 2026 18:38

@Karthik-Chowdary Karthik-Chowdary left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I walked through the registrar state transitions under cancellation, publication, discard, multiple waiters, and ID reuse. Serializing publication/removal under the owner mutex plus the identity check in waiter cleanup prevents a canceled waiter from deleting either a concurrently published value or a replacement registrar. Moving timeout ownership to the caller also aligns the gateway lookup with the existing 15-second initial Ping budget while preserving shorter upstream deadlines and bounding contexts that had none. The virtual-time coverage exercises the important lifecycle cases, including the former >3s failure. I fetched this head and independently ran go test -race ./util/registrar ./control/gateway; both packages pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants