Skip to content

feat(schema,query): DISKANN indexes, the F16 element type, and a working KNN operator - #75

Merged
albedosehen merged 1 commit into
mainfrom
feat/diskann-f16
Aug 12, 2026
Merged

albedosehen merged 1 commit into
mainfrom
feat/diskann-f16

Conversation

@albedosehen

Copy link
Copy Markdown
Collaborator

Brings the vector surface to parity with surql-rs 0.33.0, so a schema defined in TypeScript reaches the same SurrealDB 3.2 features as one defined in Rust. It also repairs three things that were already broken.

Added

IndexType.DISKANN and diskannIndex(name, field, dimension, {...}) define the on-disk ANN graph the 3.2 engine parses, with DiskAnnDistanceType for the metric. It is its own enum because the engine's DISKANN metric set neither contains nor is contained by the HNSW one, so an out-of-set metric is unrepresentable rather than merely refused. MTreeVectorType gained F16, I8, and U8, which HNSW also accepts.

The engine echoes a DISKANN index with DIST / TYPE / DEGREE / L_BUILD / ALPHA always spelled, defaults filled in even when the definition never stated them, and a float ALPHA carrying a trailing f suffix. The emitter spells the defaults, canonicalAlpha renders a whole number bare, and the parser excludes the f from its capture, so a definition compares equal to its own echo instead of re-applying on every reconcile.

mtreeIndex and diskannIndex throw on an element type the engine refuses for that kind.

Fixed

Vector search emitted SQL SurrealDB v3 refuses. The query builder rendered the bare <|k|> KNN form, which belongs to the KTree era and is a parse error on v3, so vector search in this toolkit did not work against a v3 server at all. It also accepted a distance argument and then dropped it. The operator now always carries a second argument: the exploration factor when vectorSearchIndexed set one, otherwise the metric, which defaults to COSINE. The new vectorSearchIndexed(field, vector, k, ef) renders the integer form the engine plans as a KnnScan over the field's index.

The migration diff silently dropped HNSW clauses. buildIndexSql kept its own copy of the clause order that knew only UNIQUE, full-text, and MTREE, so an HNSW index in a migration rendered as a plain DEFINE INDEX ... FIELDS ... with its dimension, metric, and EFC/M tuning gone. The diff now delegates to the schema emitter (generateIndexSql, newly exported), so the two cannot drift again.

The schema parser dropped unrecognised vector element types. extractVectorType matched a fixed set of five, so an index carrying any newer element type parsed back with no type and the next reconcile saw a difference that was not there.

Two existing tests asserted the old bare <|k|> rendering and are updated to the corrected form.

Note

This toolkit emits FIELDS where the Rust, Python, and Go ports emit COLUMNS. That is the existing convention here for every index kind and SurrealDB accepts both, so it is left alone; the rendered DISKANN statement is otherwise byte-identical across all four.

Verification

232 passed (1907 steps), up from a 226-passed baseline. deno lint and deno check mod.ts clean. The 5 remaining failures are the pre-existing live-database tests (CLI ping and the four integration suites); they fail identically on a clean checkout of main, verified by stashing.

Mirrors the vector surface surql-rs 0.33.0 carries, so a schema defined in
TypeScript reaches the same SurrealDB 3.2 features as one defined in Rust.

DISKANN keeps its ANN graph on disk, which suits a corpus that outgrows the
memory an HNSW graph needs. IndexType.DISKANN and diskannIndex define it,
DiskAnnDistanceType carries its metric set, and MTreeVectorType gains F16, I8,
and U8. The emitter always spells DIST / TYPE / DEGREE / L_BUILD / ALPHA and
canonicalAlpha renders a whole number bare, because the engine fills those
defaults in when it echoes the index back and a definition that omitted one
would re-apply on every reconcile. The parser excludes the engine's trailing f
suffix from its ALPHA capture for the same reason. mtreeIndex and diskannIndex
throw on an element type the engine refuses for that kind.

Three things were already broken and are fixed here.

Vector search emitted the bare <|k|> KNN form, which belongs to the KTree era
and is a parse error on v3, so it did not work against a v3 server at all. It
also took a distance argument and dropped it. The operator now always carries a
second argument: the exploration factor when vectorSearchIndexed set one,
otherwise the metric, defaulting to COSINE. vectorSearchIndexed is new and
renders the integer form the engine plans as a KnnScan over the field's index.

The migration diff kept its own copy of the clause order that knew only UNIQUE,
full-text, and MTREE, so an HNSW index in a migration rendered as a plain index
with its dimension, metric, and tuning gone. It now delegates to the schema
emitter, which is exported for the purpose, so the two cannot drift again.

The schema parser matched a fixed set of five element types, so an index
carrying any other one parsed back with no type and the next reconcile saw a
difference that was not there.
@albedosehen
albedosehen merged commit 6a2fa43 into main Aug 12, 2026
11 checks passed
@albedosehen
albedosehen deleted the feat/diskann-f16 branch August 12, 2026 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant