cat-file: handle packs disappearing during concurrent repacks - #2220
Open
newren wants to merge 3 commits into
Open
cat-file: handle packs disappearing during concurrent repacks#2220newren wants to merge 3 commits into
newren wants to merge 3 commits into
Conversation
"git cat-file --batch-all-objects" opens each pack's .idx lazily, as its walk reaches that pack. If a concurrent repack removes a pack first, opening its index fails. The object database already turns this into a non-zero return from odb_for_each_object_ext(), but batch_each_object() discarded it and we reported success -- silently omitting the vanished pack's objects while still exiting 0, which is dangerous for tooling that trusts the listing to be complete. Propagate the error: return it from batch_each_object() and, when set, exit non-zero with a diagnostic instead of pretending a truncated listing was whole. This intentionally changes an existing expectation in t5313, which treats a corrupt pack index as a successful empty enumeration. A rejected index also prevents a complete listing, so reporting success there has the same problem as silently skipping a concurrently removed pack. Instead, expect failure while continuing to check that the corrupt index produces no object output. To reproduce, loop "git repack -adq" in one process while another loops "git cat-file --batch-all-objects --batch-check"; before this change the reader occasionally stopped with status 0 mid-repack, and now it errors instead. Assisted-by: Claude Opus 4.8 Signed-off-by: Elijah Newren <newren@gmail.com>
"git cat-file --batch-all-objects --unordered" reads each object from the pack and offset the walk handed it, rather than looking it up again. If a concurrent repack removes that pack in between, packed_object_info() calls use_pack(), which cannot reopen the unlinked pack and dies with "packfile ... cannot be accessed" -- even though the object still lives in the pack repack just wrote. Guard the read with is_pack_valid(): if the pack can no longer be opened, clear the pointer so we fall back to a normal object-database lookup, which finds the object in whatever pack now holds it. is_pack_valid() opens and pins the pack's fd, so once it succeeds the pack stays readable even if it is unlinked an instant later. Assisted-by: Claude Opus 4.8 Signed-off-by: Elijah Newren <newren@gmail.com>
The previous commit made "git cat-file --batch-all-objects" error out when a concurrent repack removes a pack mid-walk. That is correct but pessimistic on servers that repack often. Reduce how often it triggers by opening every pack index up front, before the walk. An index mmap survives unlink() of the .idx and pack fd pressure (close_one_pack() closes only the pack fd), so the object set is fixed once the indexes are open. For --batch-all-objects this adds no work: the walk opens every index anyway. Like f6b2625 (fsck: snapshot default refs before object walk, 2026-01-09), this narrows rather than closes the race; the residue is still caught by the previous commit rather than silently dropped. Assisted-by: Claude Opus 4.8 Signed-off-by: Elijah Newren <newren@gmail.com>
newren
force-pushed
the
en/cat-file-pack-race
branch
from
September 9, 2026 14:47
2b8d54b to
ac181a3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
git cat-file --batch-all-objectscan race with a concurrent repackthat removes a pack while object enumeration is in progress. Depending
on timing, cat-file can silently return a truncated listing with status
0 or die while trying to read an object from the vanished pack.
This series makes enumeration failures visible, falls back to a normal
OID lookup when an object has moved to a replacement pack, and opens all
pack indexes before starting the walk. The up-front snapshot greatly
narrows the race, while any remaining failure is reported instead of
being mistaken for a complete object listing.