feat(plugin): demand ladder — visit-driven render cadence, dry-run by default (v0.35.0)

[harper-joseph]

Stacked on #72 (feat/queue-watermark) — review that first; this PR's diff is only the demand-ladder commit.

What

Moves a target's render interval up or down a fixed rung ladder based on whether bots actually visit it, inside the same total render budget. Hot pages get a tighter freshness bound; pages nothing crawls get a looser one.

Why these rungs

Each rung is an availability-error budget, because availability is the only measured drift that is time-proportional — ~0.04%/hour, and directionally InStock → OutOfStock (the cache claiming stock for sold-through items):

rung	wrong-availability
6h	~0.24%
12h	~0.5%
24h	~1%
48h	~2%

• Bottoms out at 6h, not 1h. 1h buys ~0.04% for 6× the render cost — the worst trade on the curve, and the fast rungs are exactly where a runaway hot set becomes unaffordable.
• Capped at 48h. A slower rung (7d ≈ 7% wrong-availability) trades away more than the budget it frees.
• Price is not addressed and does not pretend to be. It steps at promotional events (most of the catalog at once), so no affordable interval bounds it.

Why promotion is harder than demotion

Promoting on "visited at all this interval" settles at rendering twice per visit: a window of length T contains a visit with probability ~T/P, so promote/demote balance at T = P/2. Requiring a visit in each of the last promoteWindows windows of the candidate interval asks whether a render at the faster rung would actually have been seen, and settles near one render per visit.

Why a Bloom ring and not lastVisitedAt

A timestamp column is free to read (the reschedule path already reads Target) but does not survive the traffic this exists for: at the ~10M bot-requests/day the search-bot ramp is sized for, per-URL writes land at roughly the distinct-URL rate against a replicated table whose patch path has already caused one replication incident. The ring is one merged row per node per flush and constant memory.

The error direction is the safe one: false positives render a page more often than needed (wasted work, never staleness), and there are no false negatives, so a genuinely visited page can never be demoted for lack of evidence.

Trade-off worth knowing: a Bloom filter cannot enumerate its members, so nothing here can produce a list of hot URLs. That is why cadence is adjusted per-target at reschedule time rather than driving a sweep.

Safety

• Opt in twice. enabled defaults off; dryRun defaults on. Enabling alone only computes and logs the counterfactual level distribution — a week of that reports the steady-state render budget before paying for it.
• Cold-union guard. A cold read-side union reads as "nobody visited anything", which would demote the entire corpus on the first pass after a restart. The ladder holds instead, and kicks the refresh explicitly (the lazy refresh lives inside the call that branch skips).
• maxFastFraction backstop + level histogram. Cost is not self-limiting — it scales with the hot fraction, ~0.5% of the corpus pre-ramp and rising with search-bot traffic. The histogram is the early warning that "promote the hot set" is turning into "halve every interval".
• Write discipline. Target.demandInterval is persisted only when the rung actually moves, so a converged corpus pays no extra write on the highest-volume path in the system.

Files

• render.demand.* config group (configSchema is the source of truth)
• util/visitFilter.js — Bloom ring; per-thread → per-node row → merged read, mirroring util/crawlStats.js
• util/demandLadder.js — rung arithmetic, dry run, histogram, backstop; visit probe is injected so the decision logic is unit-testable without a warm ring
• Target.demandInterval + VisitFilter table

Testing

12 new unit tests covering demote/promote walks, the base-as-ceiling rule, the candidate-interval promotion window, dry-run reporting-without-acting, cold-union hold, ladder normalization, and inert defaults.

Full suite: 455 tests / 441 pass / 14 fail — the 14 are pre-existing on the base branch (sitemap.test.js, upstream.test.js, ERR_MODULE_NOT_FOUND from deps missing in a fresh worktree); base measures 443/429/14.

Not yet wired: config validation that sliceMs ≤ fastest rung and slices ≥ promoteWindows × slowest rung. Worth adding before dryRun is turned off.

[gemini-code-assist]

Code Review

This pull request introduces a demand-driven cadence ('demand ladder') that dynamically adjusts page render intervals based on bot visit activity tracked via a Bloom filter ring. The reviewer feedback highlights several important improvements, including preventing out-of-bounds index accesses when bitsPerSlice is not a multiple of 8, avoiding event loop starvation by periodically yielding during database cursor iteration, optimizing performance by caching normalized rungs, and ensuring live-reloadable configuration changes correctly update or stop the active statistics timer.

[gemini-code-assist]

If bitsPerSlice is configured to a value that is not a multiple of 8, bitCount() >>> 3 will round down, resulting in a Uint8Array that is too small. This causes out-of-bounds index accesses when setting or checking bits at the end of the slice. Allocating (bitCount() + 7) >>> 3 bytes ensures there is always enough space for all bits.

Suggested change

	const newSlice = () => new Uint8Array(bitCount() >>> 3);
	const newSlice = () => new Uint8Array((bitCount() + 7) >>> 3);

[gemini-code-assist]

Align the length check with the corrected slice allocation size of (bitCount() + 7) >>> 3 to ensure correct validation when bitsPerSlice is not a multiple of 8.

Suggested change

	const bytes = merged.get(s);
	if (bytes && bytes.length === ((bitCount() + 7) >>> 3) && hasBits(bytes, idx, k)) return true;

[gemini-code-assist]

Import setImmediate from node:timers/promises to support yielding the event loop during cursor iteration.

Suggested change

	import { config, onConfigApplied } from '../config.js';
	import { getMutex } from './coordination.js';
	import { fnv1a32 } from './hash.js';
	import { setImmediate } from 'node:timers/promises';
	import { config, onConfigApplied } from '../config.js';
	import { getMutex } from './coordination.js';
	import { fnv1a32 } from './hash.js';

[gemini-code-assist]

When iterating over database cursors in Node.js, it is important to periodically yield the event loop to prevent starvation, especially in workers serving traffic. Unconditionally yield the event loop (using setImmediate from node:timers/promises) every config.scan.yieldEvery rows.

export async function refreshMerged(nowMs = Date.now()) {
	const VisitFilter = table();
	const newest = slotOf(nowMs);
	const oldest = newest - sliceCount() + 1;
	const next = new Map();
	let scanned = 0;
	const yieldEvery = config.scan?.yieldEvery ?? 200;
	for await (const row of VisitFilter.search({
		conditions: [{ attribute: 'slot', comparator: 'greater_than_equal', value: oldest }],
		select: ['slot', 'bits'],
	})) {
		scanned++;
		if (scanned % yieldEvery === 0) {
			await setImmediate();
		}
		if (!row?.bits) continue;
		const cur = next.get(row.slot);
		const bits = new Uint8Array(row.bits);
		if (!cur) next.set(row.slot, bits);
		else if (cur.length === bits.length) for (let i = 0; i < cur.length; i++) cur[i] |= bits[i];
	}
	merged = next;
	mergedAt = nowMs;
	return merged;
}

References

1. When iterating over large database cursors or registries in Node.js, yield the event loop (e.g., using setImmediate) unconditionally based on the number of rows scanned, rather than conditionally on writes or specific branches. This prevents event loop starvation, as awaiting on cursors only drains microtasks. Use a consistent count-based trigger across the codebase.

[gemini-code-assist]

decideInterval is called on the reschedule path (~20x/s). Calling rungs() on every single decision filters, de-duplicates, and sorts the ladder array repeatedly. We can optimize this by caching the normalized rungs and updating them only when the configuration is applied.

import { config, onConfigApplied } from '../config.js';
import { visitedWithin, visitedInEachWindow, mergedReady, ensureMerged } from './visitFilter.js';

let cachedRungs = [];

const updateRungs = () => {
	cachedRungs = [...new Set((config.render.demand.ladder ?? []).filter((n) => Number.isFinite(n) && n > 0))].sort((a, b) => a - b);
};

// Initialize
updateRungs();

/** Rungs, ascending, normalized and de-duplicated. Config is the source of truth. */
export const rungs = () => cachedRungs;

[gemini-code-assist]

The statsInterval and enabled configuration options are live-reloadable, but the running statsTimer is never updated or stopped when these options change. Adding an onConfigApplied listener ensures that the timer is correctly re-armed or stopped, and the cached rungs are updated.

Suggested change

	export function stopDemandStats() {
	if (statsTimer) clearInterval(statsTimer);
	statsTimer = null;
	armedStatsInterval = null;
	}
	export function stopDemandStats() {
	if (statsTimer) clearInterval(statsTimer);
	statsTimer = null;
	armedStatsInterval = null;
	}
	onConfigApplied(() => {
	updateRungs();
	if (!statsTimer) return;
	if (!config.render.demand.enabled) {
	stopDemandStats();
	return;
	}
	if (config.render.demand.statsInterval !== armedStatsInterval) {
	stopDemandStats();
	armDemandStats();
	}
	});