Make the master's namespace durable and stop silent data loss

[ChiragJS]

Live testing of the cluster turned up four failure modes that all reported success while losing or corrupting data. This fixes them, plus the higher-severity issues found alongside, and adds the test suite the repo was missing.

Every fix below is pinned by a test that I verified fails against the old behaviour.

The four confirmed data-loss bugs

	Symptom	Fix
Master restart orphaned every file	Chunks intact on both disks, file does not exist forever	Write-ahead log + checkpoints
get twice appended	5,000,000 B → 10,000,000 B, "Download successful"	Temp file + atomic rename
get with no reachable replica "succeeded"	Exit code 0, empty output directory	Propagate errors
put wrote 1 replica, master recorded 2	One data dir empty, metadata claimed both	Write all replicas synchronously

Metadata durability

The master kept its namespace only in Go maps. A restart left the data on disk but permanently unreachable, and the stranded chunks were never reclaimed — registration would resurrect them into ms.Chunks with an empty FileName, after which nothing ever collected them.

This now follows the GFS operation-log design (what HDFS calls edits + fsimage), with metadata split into two classes:

	Stored where	Recovered how
Namespace (file → ordered chunk list)	WAL + checkpoints under -meta	Replayed at startup
Chunk locations	Nowhere — deliberately	Rebuilt from registration + heartbeats

Namespace mutations are fsync'd before the RPC is acknowledged, so an acknowledged write survives a crash. Checkpoints are published by atomic rename and only bound replay time; the log stays the source of truth. A torn trailing record is discarded on recovery — it was never fsync'd, so it was never acknowledged.

Locations are intentionally not persisted: chunkservers are the only authority on what they actually hold, so storing locations would just yield a stale copy that disagrees with the disks. That is also why a restarted master waits out -orphan-grace before deleting chunks it has no metadata for.

The other three

• Append on re-download — O_APPEND was applied to chunk 0 as well as 1..N. Downloads now stream into a temp file and rename into place, so a re-run replaces the previous copy and an interrupted run leaves nothing behind.
• Silent success — the replica loop continued past every failure and fell through to return nil. Both directions now propagate errors, and the chunker distinguishes io.EOF from a real read failure instead of treating every error as end-of-file (which silently truncated uploads).
• Single-replica writes — the uploader breaks on first success. Background replication did heal it, but left a window at RF=1 that the master could not see. The client now writes every allocated replica before reporting success.

Also fixed

• Master stalled the cluster on one slow chunkserver. ms.mu was held across the blocking stream.Send in Heartbeat, while chunkservers blocked their stream reader when task queues filled (// its fine for now). The response is now built under the lock and sent outside it, and full queues drop tasks rather than back-pressure the stream — the master recomputes outstanding work each heartbeat, so a dropped task is retried, not lost.
• Heartbeat reconciliation rewritten to derive state from what a chunkserver reports rather than assuming dispatched tasks succeeded. Over-replication victims are picked deterministically, so concurrent heartbeats can't each be told to delete their copy.
• Three busy-spin loops where an error continued past the sleep; reconnects use exponential backoff with jitter.
• disk_usage() measured a hardcoded /home, not the storage directory — so co-located chunkservers reported identical figures and storage-aware placement degenerated to an arbitrary choice.
• Partial chunks advertised as complete. Uploads landed on the final filename while the directory scan ran every 10s. They now commit via fsync + rename, with .partial- files ignored and cleaned up at startup.
• Nil client cached on a failed dial in the uploader — a latent panic.
• Path traversal: chunkId is attacker-controlled and was joined straight onto a path. Now rejected.
• Fatal startup errors exited 0. I hit this during testing — a master that failed to bind still looked like a clean shutdown to any supervisor.
• GracefulStop could never complete, because heartbeat streams never end. Shutdown now falls back to a forced stop.
• gRPC status codes (NotFound, AlreadyExists, ResourceExhausted, …) instead of bare fmt.Errorf, so clients can tell failures apart.
• GetFileInfo returns chunks in allocation order, removing the client's parse-IDs-and-sort step.
• Master flags (-port, -meta, -replication-factor, -live-threshold, -orphan-grace) — it was the only binary with none.

Tests

The repo had zero tests; that was the root cause of most of the above. Added ~40 across five suites — unit tests for the metastore, master reconciliation, chunker, and path validation, plus end-to-end tests over real gRPC covering all four data-loss bugs.

ok  dfs/internal/chunkserver          ok  dfs/internal/master
ok  dfs/internal/client/uploader      ok  dfs/internal/master/metastore
ok  dfs/internal/integration

Clean under -race. go test -short skips anything that binds a port. Also adds CI (build, vet, gofmt, -race) and a Makefile.

Not in this PR

Deliberately left for follow-ups, and now recorded under "Known Limitations" in the README: checksums (the protocol field exists but is always zero, so bitrot is undetectable — there's an existing checksum branch), client RPC deadlines, delete/list/overwrite, TLS/auth, and opaque chunk IDs.

Reviewing

internal/master/metastore/ and the reconcile rewrite in internal/master/server.go are the parts worth the closest look — everything else is comparatively mechanical.

🤖 Generated with Claude Code