Skip to content

perf(hnsw-rabitq): optimize HNSW RaBitQ index building - #755

Merged
egolearner merged 13 commits into
alibaba:mainfrom
JalinWang:exp/hnsw-rabitq-optimize-performance
Sep 21, 2026
Merged

egolearner merged 13 commits into
alibaba:mainfrom
JalinWang:exp/hnsw-rabitq-optimize-performance

Conversation

@JalinWang

@JalinWang JalinWang commented Sep 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Reuse fetched raw vectors during HNSW neighbor pruning instead of repeatedly reading the provider in O(M²) distance checks.
  • Batch FLAT provider key lookups so each neighbor batch acquires the shared key-map lock once.

Performance

100K fixed-seed random FP32 vectors, 1024 dimensions, Cosine, M=16, ef_construction=200, 16 threads.

Version Optimize time Improvement
baseline (44d8bc6) 600.1 s
reuse pruning vectors 129.5 s 4.63×
+ batched provider lookup 49.7 s 2.61× additional / 12.1× overall

Final implementation scales from 365.4 s at 1 thread to 49.7 s at 16 threads (7.35×).

Testing

  • flat_streamer_test
  • hnsw_rabitq_streamer_test
  • CollectionTest.Feature_Optimize_HNSW_RABITQ

@JalinWang JalinWang changed the title perf: optimize HNSW RaBitQ index building perf(hnsw-rabitq): optimize HNSW RaBitQ index building Sep 15, 2026
@JalinWang
JalinWang marked this pull request as ready for review September 15, 2026 08:59
@JalinWang
JalinWang requested review from egolearner and removed request for chinaux September 15, 2026 09:04

@egolearner egolearner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

很赞的结果!

Comment thread src/include/zvec/core/framework/index_provider.h

@egolearner egolearner left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@egolearner
egolearner merged commit 7bd50f8 into alibaba:main Sep 21, 2026
36 of 37 checks passed
@JalinWang
JalinWang deleted the exp/hnsw-rabitq-optimize-performance branch September 21, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants