[Tracking][Phase II] Improve optimize()/compaction performance and resource efficiency

[JalinWang]

Background

Phase I (#98) introduced a safe segment-reuse path for compaction:

• Reuse the first compatible vector index as the merge base.
• Append tail segments instead of rebuilding the complete index.
• Support FLAT, HNSW, and HNSW_RABITQ when there is no filter and schemas are compatible.
• Fall back to a full rebuild for filters, incompatible schemas, IVF, VAMANA, and other unsafe cases.

This significantly reduced compaction time for reusable paths. Phase II will build on that work and address the remaining rebuild, file-copy, memory, and I/O costs.

Goals

• Make compaction cost scale primarily with the changed/tail data where possible.
• Reduce write amplification, temporary disk usage, and peak RSS.
• Expand incremental/reuse support to more index and compaction scenarios.

Tracking fixes and improvments here.

[JalinWang]

Memory footprint:

• IVF: ↓60% perf(ivf): reduce peak memory usage during index build and merge #733
• DiskANN：↓65% refactor(diskann): adapt diskann to turbo quantizer #678

[JalinWang]

Index build time:

• Hnsw RabitQ: 12.1× faster perf(hnsw-rabitq): optimize HNSW RaBitQ index building #755