feat(plugin): bulk cache invalidation — one row, compared on the serve path, instead of rewriting 1.6M rows (v0.35.0)

[harper-joseph]

Stacked on #72 — review that first, or read this diff against feat/queue-watermark. GitHub is showing #72's commits here because it is not merged yet; this PR's own work is the last three commits.

"Everything of this kind is wrong as of now; stop serving it." One row records a scope and an instant; from then on any cached page in that scope rendered before that instant stops being served, and bots get the origin until the page re-renders on its normal cadence.

The mechanism is a comparison, not a rewrite

That is the whole design, and it is a measured choice. The tree already has an invalidation mechanism — Target.revalidate's collection form — and it rewrites the corpus. Measured on 400,000 rows, one node's slice of a 1.61M-key corpus:

	rewrite the corpus	record an epoch
wall time	15.7 ± 0.4 s	0.18 ms
audit written	61.8 MB per node, 162 B/row	102 bytes
schedule/page writes	800,000	0
undo	impossible	delete the row

Pacing does not fix the rewrite — batch-100-with-yields kept the same 162 B/write, took 8.9× longer, and made claim's max latency worse (47.4 → 62.4 ms). And the naive form is uniquely bad: collapsing every due time to currentMinuteMs() piles the rows exactly where the claim scan seeks, taking it 0.36 ms → 11.59 ms (32×), and that only clears on the next compaction of the store, which needs write pressure.

Consulting the epoch costs ~13 µs of storage time. The read is 0.107 ms p50 over a socket whose transport floor is 0.094 ms, against an existing hot page read of 0.291 ms p50 that also streams a ~50 KB blob. At ~4,000 bot requests/day, of which ~2,900 are cache-servable, the whole feature's steady-state cost is ≤5,800 point reads/day.

Why there is no sweep, and never will be

1,530,046 PDP keys ÷ a measured fleet ceiling of 71,289 renders/hr = a 21.5 h floor at 100% utilisation — against the 48 h such a page waits anyway — while measured utilisation is already 98% with a 3.05 h standing backlog. A sweep would also displace the 1 h and 12 h routes behind 1.53M PDP keys, because claim is strictly due-time-ascending.

Cadence-heal is correct by construction, not by preference: nothing in claim, syncQueueState or reconcile reads PrerenderedPage, so the epoch cannot perturb the queue at all. invalidation.reenqueue (default off) optionally pulls forward the pages bots actually crawl — request-triggered, no timer, no scan.

The scope axis is a closed set, and that is load-bearing

all, or one prerender route written route:<match>:<path> exactly as ingress.routes declares it. classifyPath already returns the matched entry, so a page's scope derives synchronously with no extra read.

Two reasons this had to be closed rather than free text, both about silence:

1. A scope that matches nothing is the worst failure available, because the operator's mitigation appears to have worked. A free-text prefix cannot be validated, so one typo records a row that reports green and demotes nothing. Against a closed set a typo is a 400 listing the valid literals.
2. It is what keeps the read affordable. A request matches exactly one route, so resolution is all plus at most one route key — two point reads by known key, never a walk. A prefix axis turns resolution into a scan and forces a refresh timer back into the design.

Sub-route granularity costs zero code: declare a narrower route, which also buys that path its own metrics series and its own renderInterval.

Precedence between overlapping scopes is max(invalidatedAt) — not most-specific-wins, which reads as the natural rule and is the one rule that can silently serve invalidated content: a leftover rehearsal row would hide a fresh all, and no coverage check can catch that because coverage enumerates routes, not competing rows.

How the epoch reaches every worker: it doesn't

There is no propagation mechanism, deliberately. Every worker resolves per request, so apply and undo are effective on the next request everywhere, and "the epoch never reached worker 5" is not a state this design can be in. A refresh timer was costed and rejected: 8 workers × 4 nodes × 1/min is 46,080 reads/day against ≤5,800 for per-request resolution — ~7.9× more storage work, while introducing that very failure mode and delaying both apply and undo by an interval.

The affordability comes from the gate: the epoch is read only when the request would otherwise have been a cache serve. A miss, a cache skip, a non-GET, a page past its SWR window, and enabled: false all pay zero. The gate is also what makes the invalidated counter mean "cache serves this invalidation is costing us" rather than a tally of every stale key in the scope.

Things that would have silently defeated it

• A conditional GET. The validators a crawler holds came from us, off the snapshot just invalidated. They are forwarded to the origin, whose ETag may be a publish date rather than a content hash → 304; and applyConditional compares them against the response headers, which on an origin proxy are the origin's → 304 locally. Either way the request records cacheStatus: 'invalidated' while the crawler keeps the pre-change bytes. Every signal reads as success. Both halves are closed, scoped to that one verdict. This is not the "the edge keeps its own TTL" caveat — a TTL expires, a 304 loop does not.
• renderNow.fallback: 'stale' re-read and served a cached page with no freshness check at all. It is opt-in staleness, so no freshness check belongs there — but an invalidation says "this is WRONG", not "this is old", and that path is the one cache read that never passes the gate.
• An invalidated-but-still-fresh page being treated as a miss. With defaultMissMode: 'prerender', every authorized request to one became an unjittered currentMinuteMs() schedule write plus up to 30 s of polling for a render with no reason to arrive. Forced to 'origin' unless prerender was explicitly asked for — hence missModeExplicit, since missMode alone cannot tell a gesture from a default.
• A route renamed by a live config edit, which un-invalidates whatever that scope covered and leaves a row that still looks applied. collectConfigWarnings cannot detect it — it is synchronous and cannot read a table — so the resolvability check runs at boot and on every config apply.
• enabled: false while rows exist. Reported on boot, on every apply, and flagged on GET /invalidations.

Undo is asymmetric, by construction

Clearing restores service on the next request for every page still inside its own expiry/SWR window, and cannot restore a page whose window elapsed while the invalidation was active — its lifetime ended on its own terms and nothing here rewrote lastCached. So a long-running invalidation cannot be fully undone. The clear response says so with the numbers attached, rather than reporting an unqualified success.

API

POST /prerender_admin/invalidate  {"scope":"all","reason":"…","dryRun":true}   # preview
POST /prerender_admin/invalidate  {"scope":"route:prefix:/catalog/","reason":"…"}
GET  /prerender_admin/invalidations
POST /prerender_admin/invalidate  {"scope":"all","mode":null}                  # clear

dryRun returns exactly the body the real call returns, minus the write, so "what would this do" and "what did this do" cannot drift apart. invalidatedAt is server-stamped and a caller-supplied one is rejected, not ignored — an operator who believes they backdated an invalidation and did not has a corpus they think is invalidated and isn't.

Bugs in existing code, fixed here

• Both fromSitemap clobbers. put REPLACES the record, so an on-demand render of a sitemap-listed URL cleared the flag, and the renderer then skips serializing a non-indexable sitemap-listed page — that page silently stops being cached at all. The Target.revalidate one was worse than expected: it trusted the caller's projection, so ?select(url) made sitemapUrl absent and indistinguishable from null, and every revalidated key re-filed fromSitemap: false; a select without url made the sweep report success having written nothing. A source-scan test now fails the build on any literal fromSitemap: false, which is the shape this bug has taken twice.
• page_age_negative. Negative ages were silently discarded on every served request, and they are the only evidence anywhere of cross-node clock skew — which is half of what invalidation.pad covers, and the half whose default rests on no evidence.
• Debug headers dropped by any 304, on every path, not just this feature's.

Testing

530 tests pass, lint and Prettier clean, suite run 5× to confirm no flakes.

The accelerator was implemented and then reviewed by three adversarial reviewers on distinct lenses (queue safety, device-pair/jitter alignment, guard completeness). 12 findings: 8 real and fixed, 4 rejected. The two worth naming were both defects in the first implementation — a per-URL verdict derived from one device's row (on a split pair, a mobile crawl re-armed a desktop row a desktop crawl had just been refused), and a rate window that rolled in both directions and sampled the minute before an awaited read, so the documented ceiling bounded nothing.

Writing the core-module tests found a real bug too: lkgMaxAge: 0 is documented as "fail open on the first read error", but age <= maxAge is true when both are 0 — the common case, two reads in one millisecond — so the documented way to switch the last-known-good off did not switch it off.

Ships INERT. With the table empty every verdict equals today's, and the only behavioural difference on the serve path is one conditional point read. reenqueue is off, so it adds zero schedule writes.

Not in this PR

• grace mode. hard only. grace is definitionally "hard, delayed by swrTtl", both end at origin-proxy-until-re-render, and its only distinct use case is a template change — where an origin proxy already serves the new template. The column exists so adding it later is not a migration.
• Target.revalidate's collection form is not retired. An unknown consumer script may call it; gating it belongs to the footgun-retirement work, not here.
• Five product decisions the design raises and code cannot settle: whether up to 48 h of origin-proxied PDP is an acceptable answer to a sitewide price change; whether the origin can absorb an all invalidation taking offload to zero (~120 extra origin requests/hour at today's volume, on bot traffic, and the phased bot-category rollout is explicitly intended to multiply it); whether grace should exist after all; whether a fleet-wide render-failure spike should auto-suspend active invalidations (I lean no — it un-invalidates without an operator asking, during an incident, in the direction of serving wrong prices); and who may invalidate, given super_user currently makes "may deploy config" and "may take 1.6M keys off the serve path" the same privilege.

Before enabling it on anything real

Rehearse on the narrowest route first — that step is the whole value of the feature being usable by someone who has done it once before, under pressure. Confirm the debug headers on a direct-to-node request, invalidated in the per-route serve counter, healing with nobody clearing the row, and that clearing restores service on the next request. Then a wider route. Then reenqueue, separately. Only then all.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a bulk cache invalidation feature that demotes cached pages rendered before a specified epoch without rewriting the corpus. It also implements a demand-driven 'invalidation accelerator' to prioritize re-rendering pages that bots actually crawl. Additionally, the changes fix several pre-existing bugs related to sitemap flags being cleared during on-demand renders and sweeps. The review feedback suggests improving robustness by adding defensive input validation guards for exported functions to prevent potential TypeErrors, and performing case-insensitive hostname comparisons to avoid mismatches.

[gemini-code-assist]

According to the general rules, hostname comparisons should be performed case-insensitively to comply with the specification and prevent mismatches caused by different DNS or server configuration casings.

Suggested change

	const owner = getResidencyByUrl(url);
	if (owner !== server.hostname) return refuse('not-owner', { owner });
	const owner = getResidencyByUrl(url);
	if (owner.toLowerCase() !== server.hostname.toLowerCase()) return refuse('not-owner', { owner });

References

1. Hostname comparisons should be performed case-insensitively to comply with the specification and prevent mismatches caused by different DNS or server configuration casings.

[harper-joseph]

Dismissing — and the suggested fix would be actively wrong here. getResidencyByUrl returns an element of [server.hostname, ...server.nodes].sort(), so when this node owns the key the returned winner is the server.hostname string itself — the self-comparison cannot miscompare on casing. A casing mismatch could only arise if this node also appeared in server.nodes under different casing, which double-enters it in the HRW ring and breaks residency ownership cluster-wide — a cluster-config bug no local .toLowerCase() repairs. Making only this one site case-insensitive would diverge from the identical exact comparison in RenderQueue and from the residency-list computation itself; consistency of the exact comparison is the correctness property.

[gemini-code-assist]

Defensive programming: invalidatedBy is a nullable reference and could potentially be null or undefined if accelerateHeal is called directly from other contexts. Adding a guard at the entry point of accelerateHeal prevents potential TypeErrors downstream (such as when accessing invalidatedBy.at on line 295).

Suggested change

	export const accelerateHeal = async ({ url, cacheKey, invalidatedBy }) => {
	const scope = invalidatedBy?.scope ?? null;
	const refuse = (outcome, extra) => {
	record(outcome, scope);
	return { outcome, ...extra };
	};
	export const accelerateHeal = async ({ url, cacheKey, invalidatedBy }) => {
	const scope = invalidatedBy?.scope ?? null;
	const refuse = (outcome, extra) => {
	record(outcome, scope);
	return { outcome, ...extra };
	};
	if (!invalidatedBy) return refuse('error');

References

1. Guard exported functions against invalid inputs (e.g., non-string types) to ensure the robustness of the module's public API.

[harper-joseph]

Resolved the underlying ambiguity the opposite way in 7c5c4f1. Null is not an input this function supports: the one production caller (maybeAccelerateHeal) returns before dispatch without a verdict, and guards 1–4 don't depend on it — a hypothetical null from a future caller surfaces as a caught, counted, logged error outcome in the detached path, never a 500 and never a write. What WAS misleading is the invalidatedBy?.scope at entry, which implied null is tolerated when the guards below don't extend that tolerance — so the ?. is now removed and a bad caller fails loudly on the first property read, rather than adding a guard that quietly accepts an input the contract rejects.

[gemini-code-assist]

Defensive programming: rows is expected to be an iterable array, but if a database read fails or a caller passes null/undefined (or a non-array), iterating over it with for...of will throw a TypeError. Adding a check to ensure rows is an array prevents potential crashes in the admin views.

Suggested change

	export const epochFromActiveSet = (rows, routeScope) => {
	if (!config.invalidation.enabled) return null;
	let winner = null;
	for (const row of rows) {
	export const epochFromActiveSet = (rows, routeScope) => {
	if (!config.invalidation.enabled || !Array.isArray(rows)) return null;
	let winner = null;
	for (const row of rows) {

References

1. Guard exported functions against invalid inputs (e.g., non-string types) to ensure the robustness of the module's public API.

[harper-joseph]

Dismissing. All four call sites guarantee an array (invalidations ?? [], (await readWithTimeout(...))?.rows ?? [] twice, and a = [] default parameter), so the guard would be dead code today — and if a future caller does pass garbage, silently returning null is the wrong failure: null here means "nothing invalidated", i.e. the guard would convert a type bug into a page reported fresh and servable in the console — the silent fail-open this module's own header names as its worst failure class. A thrown TypeError in an admin view is loud and bounded, which is the better direction for this specific function.

[harper-joseph]

Deep-review pass complete. Four findings fixed in 7c5c4f1; all three Gemini inline threads answered (each dismissed with reasoning on-thread).

• A stale row whose route was renamed away could not be cleared through the API — the closed-set check ran before the clearing branch, so the documented remediation ('clear the rows') answered 400 Unknown scope. Clearing now runs first (recording still validates; clearing a never-recorded scope still 404s). This came with the first API-level tests for invalidate(), including the renamed-route clear.
• The renderNow timeout→origin fallback forwarded the crawler's validators — the exact 304 defeat this PR closes on the direct origin path and the stale fallback. It now strips validators whenever an epoch is active for the scope.
• The 'effective on every node — nothing to propagate' responses ignored replication: each node reads its own replica, and this cluster has seen fresh rows silently fail to replicate for days. Apply/clear responses and the module comment now say so, and name the peer-verification rehearsal step. (Follow-up worth doing separately: per-node presence in GET /invalidations via the existing bounded peer fan-out.)
• invalidation.pad (2 min) didn't cover the in-flight window its own text calls 'the certain one' — a job may post back up to queue.jobLeaseTime (10 min) after claiming, stamping pre-change bytes as healed for a full render interval with every counter green. Default raised to 10 min, plus a cross-option config warning when pad < jobLeaseTime.

Also: accelerateHeal no longer half-tolerates a null verdict (?. removed at entry — see the Gemini thread for why a guard would be the wrong direction).

Untested-but-argued remainder, flagged as follow-ups rather than blockers: unit tests for stripValidators/suppressConditional (the two halves of the 304 defeat), and the five product decisions in the PR body are untouched. 534 tests, lint, Prettier green.