cache: serialize blob computation writers - #7182
Open
tonistiigi wants to merge 2 commits into
Open
tonistiigi wants to merge 2 commits into
tonistiigi wants to merge 2 commits into
Conversation
Reusing sr.ID() as the content store ingest reference makes a following
attempt collide with the previous one, because containerd releases the
ingest lock asynchronously: the gRPC proxy's Close() is a fire-and-forget
CloseSend(), and the local writer unlocks only after its fp.Sync() has
completed. So any failure of the overlay differ, which is followed by a
fallback to the containerd differ using the same reference, turns into a
hard failure:
mount callback failed on .../tmpmounts/containerd-mount...:
failed to open writer: ref moby/1/<id> locked for <d>: unavailable
This is independent of the kernel: it fires whenever the overlay differ
fails for any reason, which is why it is the more important half of the fix.
Generate a reference per attempt instead, keeping the cache record ID as a
prefix so ingests can still be traced back to the record they belong to.
Also abort the ingest when the fallback differs fail, as they do not clean
up an ingest that was opened with an explicit reference.
Additionally close the writer before aborting the ingest. Aborting while
the writer is still open always fails with "directory not empty", which
rolls back the metadata transaction removing the ingest bucket and leaks
the ingest directory on disk:
failed to abort writer "<ref>" error="unlinkat .../ingest/<digest>:
directory not empty"
This ordering is only safe once references are unique, hence both changes
in a single commit.
Signed-off-by: okhowang(王沛文) <okhowang@tencent.com>
tonistiigi
force-pushed
the
serialize-writers
branch
from
September 21, 2026 19:54
2690899 to
bac419f
Compare
Use the cache record ID as a stable ingest reference and wait for an existing writer through content.OpenWriter. Retry external differ calls on writer contention because their content writer is not directly accessible. Do not abort failed stable-reference ingests after closing them, as another waiter may already own the transaction. Truncate resumed ingests before recomputing and create fresh compressor state for every differ attempt. Also ensure the overlaybd and Windows differ paths release their writers on failure. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
tonistiigi
force-pushed
the
serialize-writers
branch
from
September 21, 2026 22:59
bac419f to
55e5212
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
replaces #7168
Instead of forcing a unique random ref like in #7168 , use
content.OpenWriterfor synchronization where possible, and copy the same logic tocompareWithRetrybecause containerdComparerdoesn't support ref synchronization atm.@okhowang