feat(schema,query): DISKANN indexes, the F16 element type, and a working KNN operator

CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## [Unreleased]
+
+### Added
+
+- **DISKANN vector indexes and the F16 element type (SurrealDB 3.2).** `IndexType.DISKANN` and `diskannIndex(name, field, dimension, { distance, vectorType, degree, lBuild, alpha, hashedVector })` define the on-disk ANN graph the 3.2 engine parses, with `DiskAnnDistanceType` for the metric (`COSINE` / `COSINE_NORMALIZED` / `EUCLIDEAN` / `INNER_PRODUCT`). It is its own enum because the engine's DISKANN metric set neither contains nor is contained by the HNSW one, so an out-of-set metric is unrepresentable rather than merely refused. `MTreeVectorType` gained `F16`, `I8`, and `U8`, which HNSW also accepts. The schema emitter, the `INFO FOR TABLE` parser, and the migration diff all carry the new form.
+
+  The engine echoes a DISKANN index with `DIST` / `TYPE` / `DEGREE` / `L_BUILD` / `ALPHA` always spelled, defaults `EUCLIDEAN` / `F32` / 64 / 100 / 1.2 filled in even when the definition never stated them, and a float `ALPHA` carrying a trailing `f` suffix (`ALPHA 1.2f`). The emitter spells the defaults, `canonicalAlpha` renders a whole number bare (`ALPHA 2`), and the parser excludes the `f` from its capture, so a definition compares equal to its own echo instead of re-applying on every reconcile.
+
+  `mtreeIndex` and `diskannIndex` throw on an element type the engine refuses for that kind: MTREE still parses only `F64` / `F32` / `I64` / `I32` / `I16`, and DISKANN accepts only `F32` / `F16` / `I8` / `U8`.
+
+- **`Query.vectorSearchIndexed(field, vector, k, ef)` reaches a vector index.** The second argument of the KNN operator decides the plan: an integer is the exploration factor and the engine answers with a `KnnScan` over the field's HNSW or DISKANN index, while a metric keyword there asks for an exhaustive `KnnTopK` over a table scan. Reach for it whenever the column carries an index; the metric belongs to the index, so the method takes none.
+
+### Fixed
+
+- **Vector search emitted SQL SurrealDB v3 refuses.** The query builder rendered the bare `<|k|>` KNN form, which belongs to the KTree era and is a parse error on v3, so vector search in this toolkit did not work against a v3 server at all. It also accepted a `distance` argument and then dropped it. The operator now always carries a second argument: the exploration factor when `vectorSearchIndexed` set one, otherwise the metric, which defaults to `COSINE` when the caller omits it.
+
+- **The migration diff silently dropped HNSW clauses.** `buildIndexSql` kept its own copy of the clause order that knew only UNIQUE, full-text, and MTREE, so an HNSW index in a migration rendered as a plain `DEFINE INDEX ... FIELDS ...` with its dimension, metric, and EFC/M tuning gone. The diff now delegates to the schema emitter (`generateIndexSql`, newly exported), so the two cannot drift again.
+
+- **The schema parser dropped unrecognised vector element types.** `extractVectorType` matched a fixed set of five, so an index carrying any newer element type parsed back with no type and the next reconcile saw a difference that was not there.
+
 ## [1.7.0] - 2026-07-29
 
 ### Added
@@ -51,7 +71,7 @@
 
 - **Edge round-trip parity in the schema parser** (`parseEdgeInfo`): edges defined via `edgeSchema` / `EdgeDefinition` now round-trip through `parseEdgeInfo` with the same fidelity tables already had. Edge mode is detected from the `DEFINE TABLE` statement — `TYPE RELATION` resolves to `EdgeMode.RELATION`, `SCHEMAFULL` to `EdgeMode.SCHEMAFULL`, anything else to `EdgeMode.SCHEMALESS` — so SCHEMAFULL edges no longer collapse into the RELATION case. `FROM <table>` and `TO <table>` are parsed independently so a malformed live definition that lost one clause surfaces as missing-endpoint drift instead of a parse failure. On `TYPE RELATION` edges the auto-emitted `in` and `out` fields SurrealDB stores are stripped on parse — they are implicit when `TYPE RELATION` is set, so the code-side `EdgeDefinition` does not declare them and round-trip diffs were flagging them as orphan additions. Per-action `PERMISSIONS` (including the comma-joined `FOR select, create, update, delete WHERE …` shape v3 emits) round-trip via the existing `parseTablePermissions` helper.
 - **`stripBrackets(value)` helper** in `src/utils/helpers.ts`, also re-exported from the package root. SurrealDB v3 wraps record-id keys that contain anything other than `[a-zA-Z_][a-zA-Z0-9_]*` or pure digits in unicode angle brackets `⟨ … ⟩` (U+27E8 / U+27E9). Downstream consumers that wanted the bare `table:id` wire shape were calling `value.replace('⟨', '').replace('⟩', '')` themselves at every API boundary; `stripBrackets` centralises that strip and also accepts the legacy ASCII `< … >` form, so consumers can drop their own ad-hoc `.replace` calls. `null` and `undefined` are passed through untouched so the helper is safe to apply unconditionally. `recordIdToString` now delegates to `stripBrackets`, picking up ASCII-bracket handling as a side benefit.
-- **Transaction-bound `upsertMany`**: the `client` argument now accepts either a `Surreal` connection (autocommit, legacy behaviour) or an active `Transaction` (atomic). In the transaction mode the same per-record `UPSERT … CONTENT { … }` statements are queued on the supplied transaction via `trx.execute`, inheriting the surrounding `BEGIN TRANSACTION` / `COMMIT TRANSACTION` framing so a single bad record rolls the *entire* batch back on commit instead of leaving the database half-seeded. The mode is auto-detected — no call-site rewrite is needed beyond passing the transaction handle. Results are not available at call time in transaction mode (`Transaction.execute` buffers); the per-row results land in `Transaction.commit()`'s return value. `upsertMany` also gains an optional `conflictFields` parameter (matching the surql-py port) — fields in this list are emitted as a `WHERE field = value AND …` clause appended to each UPSERT. The conflict values are inlined rather than parameterised because `Transaction.execute` does not bind params.
+- **Transaction-bound `upsertMany`**: the `client` argument now accepts either a `Surreal` connection (autocommit, legacy behaviour) or an active `Transaction` (atomic). In the transaction mode the same per-record `UPSERT … CONTENT { … }` statements are queued on the supplied transaction via `trx.execute`, inheriting the surrounding `BEGIN TRANSACTION` / `COMMIT TRANSACTION` framing so a single bad record rolls the _entire_ batch back on commit instead of leaving the database half-seeded. The mode is auto-detected — no call-site rewrite is needed beyond passing the transaction handle. Results are not available at call time in transaction mode (`Transaction.execute` buffers); the per-row results land in `Transaction.commit()`'s return value. `upsertMany` also gains an optional `conflictFields` parameter (matching the surql-py port) — fields in this list are emitted as a `WHERE field = value AND …` clause appended to each UPSERT. The conflict values are inlined rather than parameterised because `Transaction.execute` does not bind params.
 
 ### Fixed
 
@@ -82,7 +102,7 @@
 ### Fixed
 
 - **`Transaction.commit()` discarded the per-statement results.** `Transaction.execute()` documents that the results "become available in the value returned by `commit()`", but `commit()` returned `void`. It now flushes the `BEGIN ...; COMMIT` batch through the SDK's `query(...).responses()` accessor: it returns the per-statement results of the queued statements in order, confirms the batch actually committed, and — on a rollback — names the statement that caused it instead of surfacing only a generic "failed transaction".
-- **Table and edge `PERMISSIONS` produced un-runnable DDL.** `generateTableSql` emitted table-level permissions as a *second* `DEFINE TABLE` statement; on SurrealDB v3 a repeat `DEFINE TABLE` for an existing table fails with `The table '<name>' already exists`, and on a server that did accept it the second statement redefined the table and silently dropped its `SCHEMAFULL`/`SCHEMALESS` mode. `generateEdgeSql` ignored `EdgeDefinition.permissions` entirely, so an edge built with `withEdgePermissions(...)` lost them. Both now fold permissions into the single `DEFINE TABLE` statement.
+- **Table and edge `PERMISSIONS` produced un-runnable DDL.** `generateTableSql` emitted table-level permissions as a _second_ `DEFINE TABLE` statement; on SurrealDB v3 a repeat `DEFINE TABLE` for an existing table fails with `The table '<name>' already exists`, and on a server that did accept it the second statement redefined the table and silently dropped its `SCHEMAFULL`/`SCHEMALESS` mode. `generateEdgeSql` ignored `EdgeDefinition.permissions` entirely, so an edge built with `withEdgePermissions(...)` lost them. Both now fold permissions into the single `DEFINE TABLE` statement.
 - **`quoteValue()` flattened objects, RecordIds, and Dates with `JSON.stringify`.** A nested `SurrealFnValue`, `RecordId`, or `Date` was serialized as a JSON blob rather than SurrealQL — `{ created: <fn> }` came out as `{"created":{"__surqlFn":true,...}}`. `quoteValue()` now recurses through plain objects emitting SurrealQL object literals, renders `RecordId` instances as a record-id literal (`user:alice`), and renders `Date` instances as a `d'...'` datetime literal (a bare quoted ISO string is rejected by v3 datetime-typed fields).
 - **The migration differ emitted incomplete, mistyped DDL.** `ADD_FIELD`/`MODIFY_FIELD` diffs rendered a bare `TYPE <FieldType>`, dropping `record<target>`, array element types, and `option<...>`; `MODIFY_FIELD` only fired on a base-type change, so a changed record link or optionality went undetected. `ADD_TABLE` for a new table emitted only `DEFINE TABLE name mode;` — applying that migration created an empty table. Diffs now render field types through the shared generator, and a new table (or edge) emits its complete DDL.
 - **The schema parser could not read back the shapes SurrealDB v3 returns from `INFO FOR TABLE`**, so `diffTables` reported false-positive drift on every schema using typed, optional, record-link, or array fields. The type extractor captured only the first word after `TYPE` — `option<X>`, which v3 stores as `none | X`, parsed as `any` (losing both the type and the optionality), and `record<X>` / `array<E>` lost their inner type. A field whose name is a clause keyword (`default`, `comment`, ...) had that name mis-read as the clause. Table-level mode and `PERMISSIONS` were lost entirely, since v3 omits the table-level `DEFINE` from `INFO FOR TABLE`. The parser now unfolds `option` / `record` / `array` types and populates `recordLink` / `arrayType` / `optional`, skips the `<field>[*]` array element-spec entries, and `parseTableInfo` / `parseEdgeInfo` accept a `defineTable` argument — the `DEFINE TABLE` statement from `INFO FOR DB` — to recover table mode, permissions, and relation endpoints. `parseTablePermissions` is newly exported.

README.md

@@ -10,7 +10,7 @@ Code-first database toolkit for [SurrealDB](https://surrealdb.com/). Type-safe q
 
 - **Fluent Query Builder** — Chainable API for SELECT/INSERT/UPDATE/DELETE with full generics; `typeRecord`, `timeNow`, `mathSum`, `countIf`, `stringLower` and friends render inline in both expression and `SET` contexts.
 - **Code-first Schema + Migrations** — `DEFINE` emitters with `IF NOT EXISTS`, structured schema parser, migration runner, squash, rollback, and drift detection.
-- **Hybrid search** — MTREE/HNSW vector indexes plus full-text **BM25** (`analyzer`, `bm25Index`, `fulltextSearch`/`searchScore`) for the sparse + dense legs of retrieval; fuse the two by rank (RRF).
+- **Hybrid search** — MTREE/HNSW/DISKANN vector indexes (F16 half-precision, index-backed KNN) plus full-text **BM25** (`analyzer`, `bm25Index`, `fulltextSearch`/`searchScore`) for the sparse + dense legs of retrieval; fuse the two by rank (RRF).
 - **SurrealDB v3 correctness** — Buffered `BEGIN ... COMMIT`, unrolled `GraphQuery` depth, v3-valid `type::record()` and `FULLTEXT` indexes everywhere.
 - **`surql` CLI** — `migrate`, `schema`, `db`, `orchestrate`, `settings` subcommands (built on `@cliffy/command`).
 - **Multi-runtime** — JSR for Deno, npm for Node.js 18+.

src/migration/diff.ts

@@ -1,7 +1,7 @@
 import type { EdgeDefinition } from '../schema/edge.ts'
 import type { FieldDefinition } from '../schema/fields.ts'
-import { type IndexDefinition, IndexType, type TableDefinition } from '../schema/table.ts'
-import { fieldTypeToSql, generateEdgeSql, generateTableSql } from '../schema/sql.ts'
+import type { IndexDefinition, TableDefinition } from '../schema/table.ts'
+import { fieldTypeToSql, generateEdgeSql, generateIndexSql, generateTableSql } from '../schema/sql.ts'
 import type { BucketDefinition } from '../schema/bucket.ts'
 import { generateAlterBucketSql, generateBucketSql, generateRemoveBucketSql } from '../schema/bucket.ts'
 import { DiffOperation, type SchemaDiff } from './models.ts'
@@ -116,28 +116,16 @@ export function diffFields(
   return diffs
 }
 
+/**
+ * Render a `DEFINE INDEX` statement for a migration.
+ *
+ * Delegates to the schema emitter rather than restating the clause order. The
+ * two had already drifted: this function knew UNIQUE, full-text, and MTREE, so
+ * an HNSW index in a migration rendered as a plain index with its dimension,
+ * metric, and tuning silently dropped.
+ */
 function buildIndexSql(tableName: string, idx: IndexDefinition): string {
-  const fields = idx.fields.join(', ')
-  let sql = `DEFINE INDEX ${idx.name} ON TABLE ${tableName} FIELDS ${fields}`
-
-  switch (idx.type) {
-    case IndexType.UNIQUE:
-      sql += ' UNIQUE'
-      break
-    case IndexType.SEARCH:
-      sql += ` FULLTEXT ANALYZER ${idx.searchAnalyzer ?? 'ascii'}`
-      if (idx.bm25) sql += ' BM25'
-      if (idx.highlights) sql += ' HIGHLIGHTS'
-      break
-    case IndexType.MTREE:
-      sql += ` MTREE DIMENSION ${idx.mtreeDimension}`
-      if (idx.mtreeDistance) sql += ` DIST ${idx.mtreeDistance}`
-      if (idx.mtreeVectorType) sql += ` TYPE ${idx.mtreeVectorType}`
-      if (idx.mtreeCapacity) sql += ` CAPACITY ${idx.mtreeCapacity}`
-      break
-  }
-
-  return sql + ';'
+  return generateIndexSql(tableName, idx)
 }
 
 /**

src/query/builder.ts

@@ -37,6 +37,7 @@ interface QueryState<T> {
   readonly vectorData: readonly number[] | null
   readonly vectorK: number | null
   readonly vectorDistance: VectorDistanceType | null
+  readonly vectorEf: number | null
   readonly fulltextField: string | null
   readonly fulltextReference: number | null
   readonly fulltextQuery: string | null
@@ -65,6 +66,7 @@ function defaultState<T>(): QueryState<T> {
     vectorData: null,
     vectorK: null,
     vectorDistance: null,
+    vectorEf: null,
     fulltextField: null,
     fulltextReference: null,
     fulltextQuery: null,
@@ -181,7 +183,18 @@ export class Query<T = Record<string, unknown>> {
     return this.with({ hints: [...this.state.hints, hint] })
   }
 
-  /** Configure vector search */
+  /**
+   * Configure an exhaustive vector search, rendering the metric form
+   * `<|k,METRIC|>`.
+   *
+   * The engine plans this as a KnnTopK over a table scan: every row is
+   * compared and no index is involved. Reach for {@link vectorSearchIndexed}
+   * when the field carries an HNSW or DISKANN index.
+   *
+   * An omitted metric defaults to `COSINE`. It cannot be left out of the
+   * rendered operator: the bare `<|k|>` form belongs to the KTree era and is a
+   * parse error on SurrealDB 3.x.
+   */
   vectorSearch(
     field: string,
     vector: number[],
@@ -192,7 +205,37 @@ export class Query<T = Record<string, unknown>> {
       vectorField: field,
       vectorData: vector,
       vectorK: k,
-      vectorDistance: distance ?? null,
+      vectorDistance: distance ?? 'COSINE',
+      vectorEf: null,
+    })
+  }
+
+  /**
+   * Configure an index-backed vector search, rendering the integer exploration
+   * form `<|k,ef|>`.
+   *
+   * The second argument of the KNN operator decides the plan. An integer is
+   * the exploration factor and the engine answers with a KnnScan over the
+   * field's HNSW or DISKANN index; a metric keyword there asks for an
+   * exhaustive KnnTopK instead. The metric belongs to the index, so this
+   * method takes none.
+   *
+   * @param ef Exploration factor at query time; higher trades speed for recall
+   */
+  vectorSearchIndexed(
+    field: string,
+    vector: number[],
+    k: number = 10,
+    ef: number = 40,
+  ): Query<T> {
+    return this.with({
+      vectorField: field,
+      vectorData: vector,
+      vectorK: k,
+      vectorEf: ef,
+      // Clear the exhaustive metric so a chained call cannot leave both forms
+      // armed and quietly fall back to a table scan.
+      vectorDistance: null,
     })
   }
 
@@ -288,7 +331,15 @@ export class Query<T = Record<string, unknown>> {
     const whereParts: string[] = []
     if (this.state.vectorField && this.state.vectorData) {
       const vecStr = `[${this.state.vectorData.join(', ')}]`
-      whereParts.push(`${this.state.vectorField} <|${this.state.vectorK ?? 10}|> ${vecStr}`)
+      const k = this.state.vectorK ?? 10
+      // An integer second argument reaches the index through a KnnScan plan; a
+      // metric keyword there asks the engine for an exhaustive KnnTopK. The
+      // bare `<|k|>` form is a parse error on SurrealDB 3.x, so one of the two
+      // always renders.
+      const op = this.state.vectorEf !== null
+        ? `<|${k},${this.state.vectorEf}|>`
+        : `<|${k},${this.state.vectorDistance ?? 'COSINE'}|>`
+      whereParts.push(`${this.state.vectorField} ${op} ${vecStr}`)
     }
     if (
       this.state.fulltextField !== null &&

src/schema/mod.ts

@@ -98,6 +98,12 @@ export {
 export { generateAccessSql, generateAnalyzerSql, generateEdgeSql, generateSchemaSql, generateTableSql } from './sql.ts'
 export {
   bm25Index,
+  canonicalAlpha,
+  DISKANN_DEFAULT_ALPHA,
+  DISKANN_DEFAULT_DEGREE,
+  DISKANN_DEFAULT_L_BUILD,
+  DiskAnnDistanceType,
+  diskannIndex,
   event,
   type EventDefinition,
   HnswDistanceType,