feat(memory): surface embedding backlog via me status

[jgpruitt]

Summary

Adds a user-facing surface for async embedding progress. Embedding is fully async — creates leave embedding IS NULL, a trigger enqueues into the per-space embedding_queue, and an in-process worker pool drains it — so after a large me import … an operator had no way to see the backlog (no me status, no RPC, and worker logs are Logfire info, invisible in docker compose logs).

Implements the shared foundation + TNT-188. The shared queue_stats() function is intentionally the read-source that the TNT-193 per-space queue-depth metric will reuse.

Closes TNT-188.

What's in it

Shared foundation

• queue_stats() SQL function (packages/database/space/migrate/idempotent/003_embedding_queue.sql) — one scan of embedding_queue, {{fn}}-signature-guarded. Returns pending / in_flight / waiting / failed / oldest_pending_at. Space-wide by design (no _tree_access): the backlog is an operational property of the whole space and the caller is already gated to it by build_tree_access.
• SpaceStore.queueStats() + QueueStats type (packages/engine/space/).

TNT-188

• memory.embeddingStatus RPC on the memory endpoint (any space member; aggregate counts only, no content).
• memory.embeddingStatus() client method.
• me status command — prints server, active space, and the embedding queue counts (with a relative "oldest pending" age); --json / --yaml supported.

Design notes

• in-flight vs waiting is DB-derived from the queue row's visibility timeout (claim_embedding_batch sets vt = now() + lock_duration), so it holds across replicas without coupling the RPC to worker process state. A row orphaned by a crashed worker counts as in-flight until its lock lapses, then falls back to waiting — acceptable for a coarse status view.

Deferred to TNT-193 (not in this PR)

OTel metrics helper, RPC counter/histogram, error-by-type counter, the periodic per-space queue-depth gauge, worker/DB-pool metrics. queue_stats() is the shared read-source for that gauge.

Tests

• Engine integration: queueStats across empty / pending / claimed-in-flight / failed+completed.
• Server handler integration: memory.embeddingStatus wiring + Date → ISO serialization.
• E2e: me status text + JSON structure (count-agnostic — that suite runs a real draining worker).

Verification

• ./bun run check — 971 pass, 0 fail.
• ./bun run check:full — 1321 pass, 0 fail. E2e skipped locally (no OPENAI_KEY; CI runs it).

[Copilot]

Pull request overview

Adds an operator-facing embedding backlog “status” surface across the stack (DB → engine store → RPC → client → CLI), so users can see async embedding progress after large imports.

Changes:

• Introduces a space SQL read-source queue_stats() that aggregates embedding queue state (pending/in-flight/waiting/failed + oldest pending timestamp).
• Wires queueStats() through the space engine store and exposes it via memory.embeddingStatus RPC + client method.
• Adds me status CLI command (text + --json/--yaml) and validates behavior with integration + e2e tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
packages/database/space/migrate/idempotent/003_embedding_queue.sql	Adds queue_stats() SQL function to aggregate embedding queue metrics.
packages/engine/space/types.ts	Introduces QueueStats type for the space-level backlog snapshot.
packages/engine/space/index.ts	Re-exports QueueStats from the space engine package.
packages/engine/space/db.ts	Adds SpaceStore.queueStats() implementation calling queue_stats().
packages/engine/space/db.integration.test.ts	Tests queueStats() across empty/pending/claimed/failed/completed cases.
packages/protocol/memory.ts	Adds memory.embeddingStatus params/result schemas and types.
packages/server/rpc/memory/memory.ts	Adds memory.embeddingStatus RPC method wired to store.queueStats().
packages/server/rpc/memory/memory.integration.test.ts	Integration test covering RPC wiring and ISO timestamp serialization.
packages/client/memory.ts	Adds embeddingStatus() to the memory client namespace.
packages/cli/index.ts	Registers the new status command in the CLI.
packages/cli/commands/status.ts	Implements me status output (human + structured output formats).
e2e/cli.e2e.test.ts	Adds e2e coverage for me status (text + JSON shape/invariants).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.