feat(embedding): support dimensions for openai-compatible Matryoshka models (#405)

[GuJi08233]

Fixes #405

Summary

Add optional embedding.dimensions for the openai-compatible provider to support Matryoshka models like Qwen3-Embedding-8B (768/1024/2048/4096 via POST {dimensions}).

Omitted keeps current behavior (byte-identical identity, no re-embed). Setting triggers per-model coexistence re-embed.

Changes

• Schema packages/plugin/src/config/schema/magic-context.ts: BaseEmbeddingConfigSchema.dimensions (max 8192) + include in EmbeddingConfigSchema.transform for openai-compatible/synapse
• Identity embedding-identity.ts: getEmbeddingProviderIdentity folds ...dimensions?{dimensions}:{}
• Runtime embedding.ts / project-embedding-registry.ts: resolveEmbeddingConfig preserves dimensions; getChunkEmbeddingModelId includes dimensions; EMBEDDING_AFFECTING_KEYS adds embedding.dimensions
• Transport embedding-openai.ts: OpenAICompatibleEmbeddingProviderOptions.dimensions, this.dimensions, modelId includes dimensions, embedBatch adds ...this.dimensions?{dimensions}:{} to POST /v1/embeddings
• Schema JSON assets/magic-context.schema.json regenerated via bun build-schema.ts pattern (manual edit matching generator output)

Mirrors existing truncate/input_type/local_dtype conditional-fold pattern (#127, #155, #259).

Testing

• Local pi-magic-context@0.41.1 (Pi 0.84.4) with Qwen3-Embedding-8B via openai-compatible:
  • no dimensions → 768 (en) / 1024 (zh) dims
  • dimensions:4096 → 16384 bytes / 4096 dims (memory 2 → 0f395...), verified via memory_embeddings and embedding_registrations
  • doctor passes; build-schema guard would pass after regeneration

Checklist

• Follows truncate fold pattern (conditional spread, no global re-embed when omitted)
• Updates EMBEDDING_AFFECTING_KEYS for untrusted-load GC guard
• Regenerates assets/magic-context.schema.json

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

---

Summary by cubic

Fixes #405 by adding optional embedding.dimensions for the openai-compatible provider so Matryoshka models like Qwen3-Embedding-8B can be pinned to 768/1024/2048/4096. Omitting it keeps the current byte-identical identity; setting it sends dimensions in the POST /v1/embeddings body and folds it into the chunk identity, so changing it re-embeds existing vectors. The synapse fallback also carries dimensions so fallback requests and identities use the configured value instead of the provider default.

• Validates dimensions in the config schema with a max of 8192.
• Adds embedding.dimensions to EMBEDDING_AFFECTING_KEYS so config changes trip the GC guard on untrusted loads.
• Regenerates assets/magic-context.schema.json.

^{Written for commit 9f15915. Summary will update on new commits.}

Greptile Summary

The PR adds configurable output dimensions for OpenAI-compatible Matryoshka embedding models while preserving the existing provider-default behavior when omitted.

• Adds schema validation and generated JSON Schema coverage for embedding.dimensions.
• Includes configured dimensions in embedding identities so vector widths can coexist safely.
• Propagates dimensions through direct and Synapse fallback routing into OpenAI-compatible embedding requests.
• Treats dimensions as embedding-affecting configuration for registry and garbage-collection safeguards.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
packages/plugin/src/config/schema/magic-context.ts	Adds bounded positive-integer validation and preserves dimensions for OpenAI-compatible and Synapse configurations.
packages/plugin/src/plugin/embedding-routing.ts	Completes the prior fix by carrying Synapse dimensions into OpenAI-compatible fallback configurations.
packages/plugin/src/features/magic-context/memory/embedding-openai.ts	Normalizes dimensions, includes them in provider identity, and sends them in embedding requests.
packages/plugin/src/features/magic-context/memory/embedding-identity.ts	Incorporates explicitly configured dimensions into OpenAI-compatible provider identities.
packages/plugin/src/features/magic-context/project-embedding-registry.ts	Preserves dimensions during registry resolution and includes them in chunk-model identity.
packages/plugin/src/plugin/embedding-bootstrap-helpers.ts	Marks dimensions as embedding-affecting configuration for guarded registry updates.
assets/magic-context.schema.json	Publishes the new optional dimensions field with the same validation constraints as the source schema.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    Config[Embedding configuration] --> Schema[Schema validation]
    Schema --> Routing{Provider routing}
    Routing -->|OpenAI-compatible| Resolve[Resolved embedding config]
    Routing -->|Synapse fallback| Fallback[OpenAI-compatible fallback config]
    Fallback --> Resolve
    Resolve --> Identity[Provider and chunk identities]
    Resolve --> Provider[OpenAI-compatible provider]
    Provider --> Request[POST /v1/embeddings with optional dimensions]

Loading

_{Reviews (2): Last reviewed commit: "fix(embedding): propagate dimensions thr..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Configuration and harness compatibility

[cubic-dev-ai]

1 issue found across 7 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/plugin/src/config/schema/magic-context.ts">

<violation number="1" location="packages/plugin/src/config/schema/magic-context.ts:695">
P1: When a `synapse` embedding uses its OpenAI-compatible fallback, preserve `data.dimensions` while constructing `fallbackConfig`; otherwise fallback requests use the provider-default vector width and produce vectors that do not match the configured embedding identity.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: When a synapse embedding uses its OpenAI-compatible fallback, preserve data.dimensions while constructing fallbackConfig; otherwise fallback requests use the provider-default vector width and produce vectors that do not match the configured embedding identity.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/plugin/src/config/schema/magic-context.ts, line 695:

<comment>When a `synapse` embedding uses its OpenAI-compatible fallback, preserve `data.dimensions` while constructing `fallbackConfig`; otherwise fallback requests use the provider-default vector width and produce vectors that do not match the configured embedding identity.</comment>

<file context>
@@ -683,6 +692,7 @@ export const EmbeddingConfigSchema = BaseEmbeddingConfigSchema.transform((data)
             ...(inputType ? { input_type: inputType } : {}),
             ...(queryInputType ? { query_input_type: queryInputType } : {}),
             ...(truncate ? { truncate } : {}),
+            ...(data.dimensions ? { dimensions: data.dimensions } : {}),
             ...(data.max_input_tokens ? { max_input_tokens: data.max_input_tokens } : {}),
         };
</file context>

[magic-alfonso]

Thanks for this — the plumbing is the #127 / #155 / #259 fold, and the synapse-fallback follow-up (9f15915d) is the right fix for the greptile note. Omitted dimensions on the POST body is correct, and assets/magic-context.schema.json matches the generator.

Requesting changes before merge:

1. Omitted dimensions must not change chunkModelId. getChunkEmbeddingModelId always inserts dimensions: "" for openai-compatible configs (project-embedding-registry.ts:590). That rewrites every existing remote chunk identity on upgrade and forces a full history-chunk backfill, which contradicts the PR's byte-identical claim. Spread conditionally, the same way getEmbeddingProviderIdentity already does for this field and for truncate. Add a golden: default openai config (no dimensions) keeps today's chunk id; only a set value moves it.

2. Refuse vectors whose returned length disagrees with the request. The served-model guard still only looks at body.model. A provider that ignores dimensions will store native-width vectors under the requested-width identity; if it later honors the parameter, cosine scans mix widths and score 0 (cosine-similarity.ts:2-3). Synapse already pins width from the first embed and rejects drift (embedding-synapse.ts:846-862). openai-compatible needs the same: when dimensions is set, drop the batch if vector.length !== dimensions; pin the first observed length on the provider instance so a later drift cannot share the identity. Identity may include the requested number (so a config change rotates the space) but the gate that accepts a vector has to be the returned length.

3. Tests, red-first, on those two properties. Extend embedding-openai.test.ts "provider modelId matches canonical identity" with a dimensions case (and pass dimensions into the constructor). Add the #259-style pair: omitted keeps the golden openai identity; 4096 vs 1024 vs omitted are distinct; after a dims change, loadAllEmbeddings / chunk search under the new id do not return the old-width rows. Add a POST-body test: unset omits the key, set includes it. Add a 200-with-wrong-length test that expects nulls.

4. Regenerate the docs reference with bun packages/plugin/scripts/build-config-docs.ts. packages/plugin/scripts/build-config-docs.test.ts fails on this head.

5. Plumb dimensions through probeEmbeddingEndpoint and the dashboard Test Connection, the same way input_type / truncate already are, so doctor does not probe a different vector space than the live path.

Nits, not merge blockers once 1–4 land: drop the as unknown as { dimensions?: number } casts (config.dimensions already typechecks after an openai-compatible narrow); fix the "on either" comment in EMBEDDING_AFFECTING_KEYS. Dashboard parity can stay on the embedding prefix (no new OMITTED_BY_DESIGN row). Project config may keep dimensions as a tuning field next to model / truncate; destination stripping does not need to grow.

I am not asking this PR to invent an AFT semantic.dimensions key. AFT has none; it fingerprints discovered length. Send the OpenAI field when the user set it, and believe the length that came back.