Separate base URL for query embeddings (openai provider) #6078

Use Case

We run Hindsight against a self-hosted embedding service (Hugging Face TEI, BAAI/bge-m3) that is shared with other applications. The service runs two instances of the same model: a small one for short, latency-sensitive queries and a bulk one for document embedding.

Problem Statement

With HINDSIGHT_API_EMBEDDINGS_PROVIDER=openai, every embedding call goes to HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL. A long retain, consolidation or import-bank batch therefore queues recall queries behind it, and recall latency follows the bulk workload.

How This Feature Would Help

Recall stays fast while retain, consolidation and bank imports run, without a proxy in front of the embedding service.

Proposed Solution

An optional HINDSIGHT_API_EMBEDDINGS_OPENAI_QUERY_BASE_URL. When set, encode_query uses it, while everything else (encode, encode_documents, batching hints) keeps using HINDSIGHT_API_EMBEDDINGS_OPENAI_BASE_URL. Both endpoints must serve the same model, so initialize() compares the dimensions and fails on a mismatch. When the variable is unset, behaviour is unchanged.

We run this as a local patch on v1.0.0: a small wrapper class in engine/embeddings.py that holds two OpenAIEmbeddings backends (each with its own request concurrency bound), one config field, and unit tests for config parsing and routing. We are happy to share the diff here if it helps.

Alternatives Considered

• A reverse proxy that routes by request size: fragile, because the API does not mark which calls are queries.
• A single instance with more CPU: the bulk queue still blocks queries.

Priority

Nice to have

Additional Context

Hindsight 1.0.0 (ghcr.io/vectorize-io/hindsight), TEI 1.6 CPU, bge-m3 (1024 dimensions).