Fix gateway registration wait lifetime - #7150
david-piholyuk wants to merge 1 commit into
Conversation
Wait for a pending gateway registration within the request deadline instead of discarding its ID after three seconds. Bound gateway lookups to the initial Ping budget, and remove abandoned pending entries when their last waiter leaves. Serialize registration and waiter cleanup to prevent cancellation from deleting a completed registration. Cover delayed registration, deadline and cancellation cleanup, concurrent waiters, discard/replacement, and register/cancel races. Signed-off-by: David Piholyuk <240580814+david-piholyuk@users.noreply.github.com>
aa764c0 to
040554e
Compare
Karthik-Chowdary
left a comment
There was a problem hiding this comment.
I walked through the registrar state transitions under cancellation, publication, discard, multiple waiters, and ID reuse. Serializing publication/removal under the owner mutex plus the identity check in waiter cleanup prevents a canceled waiter from deleting either a concurrently published value or a replacement registrar. Moving timeout ownership to the caller also aligns the gateway lookup with the existing 15-second initial Ping budget while preserving shorter upstream deadlines and bounding contexts that had none. The virtual-time coverage exercises the important lifecycle cases, including the former >3s failure. I fetched this head and independently ran go test -race ./util/registrar ./control/gateway; both packages pass.
Problem
The gateway client starts its initial Ping concurrently with Control/Solve. Ping can arrive before Solve registers the gateway bridge. The registrar currently discards that pending entry after three seconds, although the initial Ping has a fifteen-second deadline. A delayed but valid registration can therefore fail as
NotFound: forwarding Ping: no such jobbefore the request's deadline.This addresses the registration-timeout failure mode discussed in #5171. It does not claim to fix every possible cause of that error.
Change
An unknown ID now reaches the lookup/request deadline rather than being discarded with a synthetic cancellation after three seconds. Actual Discard still wakes waiting callers with cancellation.
Reproduction and validation
The new
TestDelayedRegistrationreproduces the ordering with virtual time: Get begins immediately, Register occurs after four seconds, and the caller allows fifteen seconds. It fails against the old registrar and succeeds with this patch. The abandoned-lookup test also fails against the old registrar.I additionally ran a local end-to-end Buildx reproduction using an HTTP/2 proxy that forwards Ping immediately but delays only Control/Solve by four seconds. The Dockerfile uses
FROM scratchand copies one proof file; no external image pull, registry, daemon restart, or application build is involved.For the v0.33.0 comparison, both daemons were compiled from the same release source using Go 1.26.3 and CGO disabled, and run in identical container packaging. The patched end-to-end case passed twice. These are fault-injection correctness tests, not build-performance benchmarks.
Validated again after applying the patch to current upstream master:
go test -race -count=100 ./util/registrar ./control/gateway go vet ./util/registrar ./control/gateway git diff --checkTests cover delayed/missing registration, shorter request deadlines, cancellation cleanup, multiple waiters, discard/replacement, retention of registered values, and concurrent cancellation/registration. The full BuildKit integration matrix and
make validate-allhave not been run locally.