Overlap post-ingest flat-vector and graph output

[nvzm123]

Summary

This draft explores a post-ingest concurrency opportunity in the CAGRA-HNSW Lucene writer.
Once Lucene has accepted all documents for a segment, the writer has two independent outputs to
produce:

1. the flat vector payload used by Lucene, and
2. the CAGRA-derived HNSW graph and metadata.

The existing path writes those outputs serially. This change adds an opt-in coordinator that writes
the flat-vector branch on one platform thread while retaining CAGRA/HNSW construction on the caller
thread. Both branches are joined before the flush returns. The serial path remains the default.

This work starts only from data already accepted through addDocument. It makes no assumptions
about source files, document counts, document IDs, or vector availability before normal Lucene
ingestion.

Branch and review scope

This is a fork-only draft PR. Its base branch, pr2476-pr2653-integration-base, pins the locally
validated integration of the PR 2476 memory/lifecycle work and the PR 2653 bounded graph-processing
work. The public PR 2653 branch and this integrated validation branch have divergent histories, so
using that public branch directly would obscure this PR with prerequisite changes.

The intended review diff is limited to:

• post-ingest flat/graph overlap in Lucene99AcceleratedHNSWVectorsWriter;
• phase-level InfoStream instrumentation used to explain the benchmark result;
• a small coordinator for lifecycle, interruption, and dual-failure handling; and
• focused unit and persisted-index integration tests.

No branch or commit is pushed to the NVIDIA/rapidsai remote by this work.

Motivation

Phase instrumentation showed that a cold Jasper-10M one-segment flush spent approximately:

• 92.4 seconds writing the flat vector payload; and
• 108.1 seconds building/materializing/serializing the graph.

The serial flush took 200.5 seconds, almost exactly the sum of those independent branches. Once
document ingestion is complete, running them concurrently changes that portion of the critical path
from approximately flat + graph to approximately max(flat, graph).

The graph branch remains on the caller thread intentionally. cuVS resources are thread-local, and
moving graph construction to an arbitrary worker would change resource ownership and cleanup. The
stock Lucene flat-vector writer is the branch moved to the worker in this prototype.

Implementation

Opt-in behavior

The feature is enabled with:

-Dcuvs.lucene.experimentalPostIngestOverlap=true

The value is captured when a segment writer is constructed, so a later JVM property mutation cannot
change the behavior of a live writer. When absent or false, flush ordering remains serial.

Flush sequence

For the opt-in path, the writer:

1. prepares the graph field views after ingestion is complete;
2. applies Lucene's document sort mapping before concurrency begins when an index sort is present;
3. submits the flat-vector flush to a dedicated platform thread;
4. builds and serializes the CAGRA-HNSW graph on the caller thread;
5. interrupts the peer branch after caller-side failure where possible;
6. joins the worker in all success, failure, and interruption paths; and
7. preserves the primary failure, attaching a secondary failure as suppressed.

A fatal Error takes precedence over an ordinary exception even if the exception was observed
first. Caller interruption is converted to InterruptedIOException, the worker is joined, and the
caller's interrupted status is restored before returning.

Instrumentation

Per-segment phase messages cover:

• flat-vector flush;
• graph-input preparation;
• host-matrix materialization;
• CAGRA build and graph access;
• HNSW materialization, graph serialization, and metadata serialization;
• CAGRA close;
• total graph branch and total flush;
• merge, finish, and writer close phases.

The phase logger treats runtime failures from a custom InfoStream as non-authoritative telemetry
failures. Such a failure cannot replace the indexing result or mask an indexing exception.

Benchmark protocol

The promoted post-ingest measurements used:

• cold source-cache conditioning;
• current benchmark default CAGRA heuristics;
• CAGRA-HNSW graph degree 32 and intermediate graph degree 48;
• one indexing/input thread;
• one physical segment;
• no force merge, no tiered merge, and no compound file;
• efSearch=1500; and
• the same Java artifacts and native libraries within each comparison matrix.

No custom IVF-PQ heuristic is part of this change.

Results

Jasper 10M, 1536 dimensions

Times are seconds. "Indexing" is the benchmark's end-to-end CAGRA-HNSW indexing time.

Arm	Indexing	Ingest	Commit	Flush	Flat	Graph	Final fsync	Recall
Serial	265.670	60.169	205.321	200.453	92.399	108.051	0.023	97.966%
Post-ingest overlap	224.007	60.039	163.770	115.369	115.363	107.988	43.531	97.616%
Overlap + external whole-file fdatasync helper	207.058	60.047	146.827	114.825	114.819	107.100	27.136	98.239%
Overlap + external 64 MiB range helper	206.634	60.138	146.310	115.820	115.814	106.056	25.530	97.936%

Overlap alone reduced indexing time by 41.663 seconds, or 15.68% (1.186x). Overlap plus proactive
whole-file writeback reduced it by 58.612 seconds, or 22.06% (1.283x).

An independent run using the preserved PR 2476 native build reproduced the shape of the result:

• serial: 269.492 seconds;
• overlap: 224.590 seconds; and
• overlap plus external fdatasync: 206.898 and 206.877 seconds.

The latter pair averaged a 23.23% reduction from that matrix's serial baseline.

Deep1B 10M, 96 dimensions

Arm	Indexing	Outcome versus serial
Serial	38.933 s	baseline
Post-ingest overlap	39.114 s	0.46% slower; effectively neutral
Overlap + external whole-file fdatasync helper	33.765 s	13.27% faster (1.153x)

This control is important: overlap by itself is not universally faster. On Deep1B, it removed about
5.56 seconds from flush and moved about 5.35 seconds into final synchronization. The experiment
therefore identifies writeback scheduling as part of the end-to-end opportunity rather than claiming
that Java concurrency alone is sufficient on every dataset.

What this PR does not include

The proactive fdatasync mechanism above was an external benchmark helper that watched completed
flat-vector files. It proved that dirty-page writeback explains the serial tail, but it is not
production code and is not included here. Lucene's final file and directory synchronization remained
authoritative in every measured arm and was never removed.

Range-based writeback, larger flat-vector buffers, and .vex buffering did not provide a material
gain beyond whole-file writeback and are not proposed:

• 64 MiB range writeback was only 0.424 seconds faster than whole-file writeback on Jasper, which is
  noise-sized for these runs;
• a 128 KiB flat-vector buffer changed the combined Jasper result by about 0.4% and slightly
  regressed Deep1B; and
• graph-output buffering saved only tens to hundreds of milliseconds at 1M scale and was not on the
  overlapped critical path.

Correctness coverage

The new focused tests cover:

• proof that flat and graph tasks overlap;
• byte-for-byte parity of deterministic test outputs between serial and overlapped scheduling;
• flat-branch failure propagation;
• graph failure, worker interruption, and mandatory join-before-return;
• both fatal/nonfatal dual-failure orderings;
• caller interruption and interrupted-status restoration;
• normal, sorted, sparse, and multiple vector-field graph preparation;
• runtime failure from optional phase telemetry;
• a real GPU persisted-index round trip using the accelerated codec;
• one physical segment with index sorting and two independently sparse vector fields;
• CheckIndex, reopen, document-sort order, exact vector-to-document mapping, graph sizes, and
  vector-search usability; and
• logical parity between indexes produced by serial and overlapped flushes.

Validation completed on an NVIDIA L40S:

• focused overlap suite: 15 tests, zero failures/errors/skips;
• persisted GPU integration test repeated three times: passed;
• complete cuvs-lucene suite: 378 tests, zero failures and zero errors (30 pre-existing
  environment/assumption skips);
• Spotless check: passed; and
• Maven package with tests skipped after the full test run: passed.

Relationship to deterministic segment sizing

For a 100M input where four 25M physical segments are desired, an explicit caller-side setting such
as segmentSizeDocs=25_000_000 is a better API than overloading numIndexingThreads=4 to mean both
"four segments" and "four ingestion threads."

Simply mapping 25M to Lucene's maxBufferedDocs is not sufficient for pipelining. A one-thread DWPT
flush is synchronous: the call that adds document 25M blocks while that segment flushes, so document
25,000,001 is not ingested concurrently. Multiple DWPT threads make boundaries nondeterministic and
can retain several 25M buffers.

The caller-side staged design should instead use:

• segmentSizeDocs=25_000_000 to derive exact contiguous slices;
• numIndexingThreads=1 to mean actual addDocument concurrency within a slice;
• one serial ingest lane and one serial GPU-commit lane;
• maxInFlightSegments=2 for bounded retention;
• a per-staging-writer maxBufferedDocs strictly greater than its slice size, so the final
  addDocument does not trigger a synchronous flush before handoff;
• NoMergePolicy and forceMerge=0; and
• final addIndexes publication using same-filesystem hardlinks.

The benchmark harness already has a staged pipeline prototype with most of this scheduling. It still
derives partition count from numIndexingThreads; decoupling that into segmentSizeDocs is a harness
follow-up, not a codec change in this PR.

Existing evidence supports the staged shape: for default-heuristic Deep1B-100M with four physical
segments and no merge, the safe sequential schedule took 301.681 seconds and the safe staged schedule
took 252.269 seconds, a 16.38% reduction. That comparison overlaps ingestion of partition N+1 with
commit of partition N; it is distinct from the within-segment flat/graph overlap implemented here.

Known limitations

This remains a draft for several reasons:

1. The pinned Lucene 10.2 flat writer used here only reads its collected vector list during flush,
   and the persisted tests exercise that implementation. The abstract FlatVectorsWriter API does
   not state a general thread-migration/concurrent-read contract. Before upstreaming, the overlap
   should be explicitly constrained to the known implementation or supported by an owned immutable
   handoff contract.
2. The prototype creates one flat-output worker per concurrent segment flush. A production bulk API
   should inject a bounded scheduler and define a global resource budget.
3. A flat-output failure is collected after the synchronous graph branch returns. Correctness is
   preserved, but an early flat failure can waste the remainder of a long graph build.
4. Interrupt-insensitive filesystem I/O can delay failure return because the coordinator deliberately
   joins the worker before allowing writer cleanup.
5. The current opt-in is a JVM system property. A production bulk-indexing API should expose an
   explicit typed option with lifecycle ownership rather than a global property.
6. Production proactive writeback belongs in a controlled local-disk bulk-indexing API or Directory
   implementation, not in the generic codec and not in an external path-scanning helper.

Proposed next work

1. Refactor the benchmark/bulk caller to expose segmentSizeDocs, keep numIndexingThreads
   semantically independent, avoid constructing the unused whole-dataset prefetch reader on the
   partitioned path, and add exact-boundary/remainder/failure-cleanup tests.
2. Replace the global overlap property with an explicit bulk-writer option and a bounded executor.
3. Make the supported flat-writer concurrency contract explicit, or introduce an owned post-ingest
   snapshot/handoff that does not rely on an abstract implementation detail.
4. Move proactive writeback into the future local-filesystem bulk API while retaining final Lucene
   file and directory syncs unconditionally.
5. Re-run matched cold-cache Jasper-25M and Deep1B-100M matrices after those lifecycle changes,
   including one-segment and exact four-segment/no-merge arms.
6. Rebase this draft onto the final PR 2476 and PR 2653 branch topology once those dependencies are
   settled, then decide whether to split phase instrumentation from the production behavior.

Follow-up: PR-2481 native-FBIN attribution ablation

This follow-up tests a possible confound in the historical comparison: PR-2481 bundled its
native-FBIN/native-flat path with graph-processing optimizations, so a regressive FBIN path could in
principle have been hidden by a larger graph win.

The experiment used a 2x2 matrix on the same PR-2481-era source and native libraries:

• generic Lucene flat buffering versus PR-2481 bulk/native-FBIN buffering; and
• PR-2481's fixed 1,048,576-node serialization waves versus PR-2653's degree-aware 64 MiB waves.

All arms used cold source files, default IVF-PQ heuristics, graph degrees 32/48, 16 writer threads,
one indexing thread, one physical segment, no merge, and explicit stored vector IDs. Jasper-10M had
two balanced repetitions per cell; Deep1B-10M had three repetitions per cell after its initial
effects proved noise-sized.

Dataset / wave policy	Generic (s)	Native-FBIN (s)	Native delta
Jasper-10M / original	268.074 mean	249.582 mean	-6.90%
Jasper-10M / 64 MiB	265.420 mean	251.515 mean	-5.24%
Deep1B-10M / original	39.495 mean	39.066 mean	-1.09%
Deep1B-10M / 64 MiB	40.028 mean	39.563 mean	-1.16%

Findings:

1. The broad PR-2481 native-FBIN/native-flat bundle is not a hidden net regression. Its marginal
   Jasper benefit was 6.07%, and all four paired Jasper comparisons had the same beneficial sign.
   At 96d the marginal point estimate was 1.12% faster, smaller than the observed run-to-run spread,
   so it is effectively throughput-neutral.
2. This does not prove every internal sub-operation helps. The tested path bundles the FBIN source,
   direct native accumulation, custom .vec output, and owned writer lifecycle; isolating the reader
   alone would require another implementation seam.
3. The benefit is dimension-sensitive, consistent with avoiding work and memory pressure that scale
   with N * dimensions. Peak RSS was consistently lower on the native path.
4. Smaller serialization waves do not explain the newer end-to-end result. Their marginal Jasper
   effect was -0.14%. Deep's marginal point estimate was 1.31% slower, also smaller than its
   run-to-run spread. In the validated one-at-a-time serializer topology with a 256 GiB heap, the
   degree-aware 64 MiB policy did not save enough live payload to justify its extra policy and
   synchronization complexity. The prerequisite branch therefore restores PR-2481's fixed
   1,048,576-node waves; executor and correctness hardening remain independent.
5. Therefore, historical current-versus-PR-2481 timing differences should not be attributed to
   removal of a net-regressive FBIN path or to wave sizing. They require a matched comparison of
   source, heuristic, overlap/writeback, cache state, and segment topology.

The adaptive-wave implementation passed Spotless and the focused graph-equivalence tests,
including byte-for-byte serial/parallel serialization parity. It was used for this ablation, then
rejected after the scale review above. The final fork branch and its prerequisite base use the
original fixed 1,048,576-node policy.