fix(server): keep Error-phase sandbox records through the prune sweep - #3498
Open
pkhodade-NV wants to merge 1 commit into
Open
pkhodade-NV wants to merge 1 commit into
pkhodade-NV wants to merge 1 commit into
Conversation
The periodic store-vs-backend reconciliation sweep deleted any persisted sandbox not present in the driver's live backend snapshot, except for Completed and failed-main-process phases. A driver whose registry is in-process-only and never rehydrates after a restart (no persistence of its own) reports every previously-known sandbox as missing on the very first sweep after startup -- including ones already correctly, terminally marked Error by earlier crash detection -- so the sweep silently deleted them shortly after gateway restart, racing any client (GetSandbox/ListSandboxes/DeleteSandbox) working with the same sandbox in that window. Treat Error the same as the existing Completed exemption: it is already a settled, informational terminal state with no live compute resource to reclaim, so keep the durable record instead of deleting it. Signed-off-by: Prashant Khodade <pkhodade@nvidia.com> (cherry picked from commit 721a1659a822a72e76e3c0dffc6847f17129a3fc)
pkhodade-NV
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 20, 2026 15:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
reconcile_store_with_backend/prune_missing_sandboxinopenshell-server) deletes any persisted sandbox record not present in the driver's live backend snapshot, except forCompletedand failed-main-process phases.Errorby earlier crash detection.Error-phase records shortly after a gateway restart, racing any client (GetSandbox/ListSandboxes/DeleteSandbox) working with the same sandbox in that window -- some requests see the full record, others see it as already gone, depending purely on timing against the background sweep.Related Issue
No linked issue -- this is a localized correctness fix to a background reconciliation sweep's phase handling, disproving and superseding an initial hypothesis (that
GetSandboxandListSandboxesread from different backing stores) with a confirmed root cause after tracing both RPC handlers and the sweep itself.Changes
compute/mod.rs:prune_missing_sandboxnow treatsSandboxPhase::Errorthe same as the existingCompletedexemption -- both are already-settled, informational terminal states with no live compute resource left to reclaim, so the durable record is kept instead of deleted. (Stopping/Stopped/Startingsandboxes are unaffected -- they're still re-markedErrorwith aComputeResourceMissingcondition, as before.)Testing
prune_missing_sandbox_keeps_error_phase_records, mirroring the existingprune_missing_sandbox_releases_driver_resourcestest harness: puts anError-phase sandbox in the store, runs a sweep with zero grace period, asserts the record survives with its phase unchanged and no driver delete call was made.compute::module test suite passes:cargo test -p openshell-server --target x86_64-pc-windows-msvc --lib compute::.Checklist
Originally opened as GitLab MR !116 against our internal mirror; re-opened here against
windowsfor upstream review.