perf(diskann): reduce peak memory usage during index build and merge

[iaojnh]

Summary

Reduce peak RSS during DiskANN index construction by avoiding redundant vector copies and releasing build buffers earlier. Apply the build lifecycle and streaming merge approach introduced for IVF in #733.

Changes

• Stream merged inputs directly instead of materializing a complete merged vector holder.
• Replace the copied PQ training prefix with a lightweight view, and copy only selected training vectors while preserving the existing sampling order.
• Bound PQ encoding batches to a 4 MiB target and release graph vectors before encoding.
• Release the completed builder before opening the persisted index, then free input holders, converters, and cached vectors after success.
• Preserve retry behavior after dump or open failures, and propagate deferred source read errors during streaming dumps.

Validation

Linux ARM64, GCC 13.3, Release build:

• 112 tests passed, with 2 architecture-specific skips.
• Covered direct and merged builds across FP32/FP16 and L2/Cosine, resource cleanup, dump/open retries, deferred read failures, and repeated dumps.
• Verified that streaming PQ sampling produces the same sample bytes and order as the previous implementation.

Peak RSS

Measured against baseline 70f12576 using 20,000 vectors with 768 dimensions, 4 threads, graph degree 32, build list size 50, and 48 PQ chunks.

Build	Before	After	Reduction
Direct FP32	419.0 MiB	239.4 MiB	42.9%
Direct FP16	293.7 MiB	203.9 MiB	30.6%
Merge FP32	401.9 MiB	161.9 MiB	59.7%
Merge FP16	275.7 MiB	125.6 MiB	54.4%

Direct FP32 values are medians of three runs per version; other cases use one run per version. Peak RSS includes input preparation, construction, dumping, and opening the index. All 12 builds passed document-count and sampled-vector checks.

These synthetic measurements establish RSS savings for the tested workload. Limited query coverage and substantial timing variation do not establish recall or build-time equivalence.