Repository navigation
feat(schema,query): DISKANN indexes, the F16 element type, and a working KNN operator - #75
Merged
Merged
Conversation
Mirrors the vector surface surql-rs 0.33.0 carries, so a schema defined in TypeScript reaches the same SurrealDB 3.2 features as one defined in Rust. DISKANN keeps its ANN graph on disk, which suits a corpus that outgrows the memory an HNSW graph needs. IndexType.DISKANN and diskannIndex define it, DiskAnnDistanceType carries its metric set, and MTreeVectorType gains F16, I8, and U8. The emitter always spells DIST / TYPE / DEGREE / L_BUILD / ALPHA and canonicalAlpha renders a whole number bare, because the engine fills those defaults in when it echoes the index back and a definition that omitted one would re-apply on every reconcile. The parser excludes the engine's trailing f suffix from its ALPHA capture for the same reason. mtreeIndex and diskannIndex throw on an element type the engine refuses for that kind. Three things were already broken and are fixed here. Vector search emitted the bare <|k|> KNN form, which belongs to the KTree era and is a parse error on v3, so it did not work against a v3 server at all. It also took a distance argument and dropped it. The operator now always carries a second argument: the exploration factor when vectorSearchIndexed set one, otherwise the metric, defaulting to COSINE. vectorSearchIndexed is new and renders the integer form the engine plans as a KnnScan over the field's index. The migration diff kept its own copy of the clause order that knew only UNIQUE, full-text, and MTREE, so an HNSW index in a migration rendered as a plain index with its dimension, metric, and tuning gone. It now delegates to the schema emitter, which is exported for the purpose, so the two cannot drift again. The schema parser matched a fixed set of five element types, so an index carrying any other one parsed back with no type and the next reconcile saw a difference that was not there.
albedosehen
force-pushed
the
feat/diskann-f16
branch
from
August 12, 2026 16:33
0ab310e to
a7d628d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the vector surface to parity with surql-rs 0.33.0, so a schema defined in TypeScript reaches the same SurrealDB 3.2 features as one defined in Rust. It also repairs three things that were already broken.
Added
IndexType.DISKANNanddiskannIndex(name, field, dimension, {...})define the on-disk ANN graph the 3.2 engine parses, withDiskAnnDistanceTypefor the metric. It is its own enum because the engine's DISKANN metric set neither contains nor is contained by the HNSW one, so an out-of-set metric is unrepresentable rather than merely refused.MTreeVectorTypegainedF16,I8, andU8, which HNSW also accepts.The engine echoes a DISKANN index with
DIST/TYPE/DEGREE/L_BUILD/ALPHAalways spelled, defaults filled in even when the definition never stated them, and a floatALPHAcarrying a trailingfsuffix. The emitter spells the defaults,canonicalAlpharenders a whole number bare, and the parser excludes theffrom its capture, so a definition compares equal to its own echo instead of re-applying on every reconcile.mtreeIndexanddiskannIndexthrow on an element type the engine refuses for that kind.Fixed
Vector search emitted SQL SurrealDB v3 refuses. The query builder rendered the bare
<|k|>KNN form, which belongs to the KTree era and is a parse error on v3, so vector search in this toolkit did not work against a v3 server at all. It also accepted adistanceargument and then dropped it. The operator now always carries a second argument: the exploration factor whenvectorSearchIndexedset one, otherwise the metric, which defaults toCOSINE. The newvectorSearchIndexed(field, vector, k, ef)renders the integer form the engine plans as aKnnScanover the field's index.The migration diff silently dropped HNSW clauses.
buildIndexSqlkept its own copy of the clause order that knew only UNIQUE, full-text, and MTREE, so an HNSW index in a migration rendered as a plainDEFINE INDEX ... FIELDS ...with its dimension, metric, and EFC/M tuning gone. The diff now delegates to the schema emitter (generateIndexSql, newly exported), so the two cannot drift again.The schema parser dropped unrecognised vector element types.
extractVectorTypematched a fixed set of five, so an index carrying any newer element type parsed back with no type and the next reconcile saw a difference that was not there.Two existing tests asserted the old bare
<|k|>rendering and are updated to the corrected form.Note
This toolkit emits
FIELDSwhere the Rust, Python, and Go ports emitCOLUMNS. That is the existing convention here for every index kind and SurrealDB accepts both, so it is left alone; the rendered DISKANN statement is otherwise byte-identical across all four.Verification
232 passed (1907 steps), up from a 226-passed baseline.
deno lintanddeno check mod.tsclean. The 5 remaining failures are the pre-existing live-database tests (CLI ping and the four integration suites); they fail identically on a clean checkout of main, verified by stashing.