Repository navigation
racer: write-back caching with replicated durability and Azure upload - #1112
Draft
Jordan Olshevski (jveski) wants to merge 120 commits into
Draft
Jordan Olshevski (jveski) wants to merge 120 commits into
Jordan Olshevski (jveski) wants to merge 120 commits into
Conversation
…d peer operations Encode and decode the write peer operations and PeerResponse::Write, bind write replies to their request, and reject write operations in serve_peer until the write service lands.
Add racer_core::write::block_list_xml_from_ids, build block_list_xml on it, and drop the Azure origin's duplicate block ID and XML helpers.
…ritten-segment adoption
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved issues can exhaust HEAD deadlines, falsely report acknowledged objects as missing during ownership handoff, and mask origin probe errors as successful writes.
3 open issues
6 resolved since last review
Review effort: Balanced
A manifest replica displaced from page zero journaled a Delete when its role check dropped the session. Replay of Delete also drops the version's written pages, so a later restart lost copies the coordinator still counted as holders (repair does not act on a restart within the holder grace). Journal a new Forget record (kind 9) instead: it drops the session and keeps the pages pinned.
A version lost with its holder is not failed by design (its page waits for a holder) and may keep surviving pinned pages and cached copies after the key is written again. The leak check now tolerates pages of those versions by etag rather than by key, so a new version of the key no longer turns them into strays (seeds 152, 159). A faulted read of such a key may serve the lost version; it must be its acknowledged bytes rather than a panic on any 200 (seed 146).
…nnot fail the HEAD
A member that joins as (or is promoted to) a page's rank 0 holds no manifest or written page until the coordinator's background rebalance copies them, and the origin does not know the object before it uploads. Its own origin miss was an authoritative 404 (metadata) or 502 (pages). On caches with writes, after an origin miss, probe the other current owners and the recently replaced membership's owners copy-only before answering; metadata is 404 only when every probed owner misses.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The copy-only fallback retains an unused encryption reservation while peer reception reserves from the same Ciphertext budget, risking budget exhaustion under concurrent reads and blocking reads from surviving replicas.
1 open issue
3 resolved since last review
Review effort: Balanced
The test_dcp save helper used the writer's defaults (poll every second, up to an hour), so a fake that never commits an upload stalled the suite. Default the helper to a 10 ms poll and a 5 s timeout.
WriteService::next_deadline returned the TICK deadline even while a turn ran. A turn longer than TICK left it in the past, so the worker's park bound was zero and it spun until the turn finished. Report no deadline while a turn runs.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds object writes to Racer, aimed at model training checkpoints. A client write is acknowledged once the pages are fsynced on two ranked nodes; the replica count can be set from 1 to 3 per ClusterCache. When the cache has an origin, the data is uploaded to Azure Blob in the background with Put Block and a create-once Put Block List. Writes are create-once with explicit DELETE, pinned until uploaded or until the hold expires, and capped per worker; holders are repaired after a node is lost and the coordinator fails over to another node. The Go and Python SDKs gain Put/HostWrite/Persist/Delete, and there is a PyTorch DCP storage writer and reader that commits .metadata only after every shard is in the origin; the design and remaining known gaps are in designs/racer-writes.md.