Support runtime LLM model switching via server-side presets (per-bank/per-operation)

[eightHundreds]

I’m using Hindsight in a setup where different banks have different latency/cost requirements, and I wanted to switch the LLM model for a bank at runtime without restarting the server.

After tracing the current implementation, it looks like this is not supported today:

• The docs explicitly classify llm_provider / llm_model as static fields and say provider/model selection “requires presets - not yet implemented”.
• ConfigResolver.update_bank_config() rejects llm_model because _CONFIGURABLE_FIELDS excludes provider/model selection.
• There is even a test that asserts update_bank_config(..., {"llm_model": "gpt-4"}) raises ValueError.
• At execution time, the long-lived LLMProvider instances are created up front. with_config() only applies Gemini safety settings, so even a resolved config would not currently swap the underlying model/provider per request.

This means the current bank config API can change behavioral settings at runtime, but not the actual LLM provider/model.

There is also a small DX/documentation mismatch here: the HTTP/OpenAPI examples for BankConfigUpdate / BankConfigResponse still show llm_model in the examples, which makes it look supported even though the implementation rejects it.

Why this would help

Runtime model switching would be very useful for:

• per-bank cost/performance tuning (cheap-fast for one bank, high-accuracy for another)
• swapping models during provider incidents without restarting the whole service
• A/B testing or staged rollouts across banks
• using different retain/reflect models dynamically depending on the workflow
• multi-tenant setups where tenants should choose from approved presets without ever seeing credentials

Proposed direction

The docs already hint at what seems like the right solution: presets.

Instead of making raw llm_provider, llm_model, *_api_key, *_base_url bank-configurable, Hindsight could support server-defined presets such as:

{
  "llm_preset": "fast-default",
  "retain_llm_preset": "cheap-extraction",
  "reflect_llm_preset": "high-quality-reasoning"
}

Where presets are configured server-side and map to provider/model/base_url/api-key bundles. Then banks could switch models at runtime safely without exposing credentials or arbitrary infrastructure details.

That seems like a good fit for the current security model:

• safe to expose and mutate per-bank
• credentials still remain server-only
• immediate effect on subsequent requests via resolve_full_config()
• avoids letting untrusted callers point the server at arbitrary endpoints

Even if presets are not implemented soon

At minimum, it would help to align the API examples/OpenAPI examples with the actual behavior so llm_model is not shown as a valid bank-config override.

Curious if this is already on the roadmap. If not, I’d love to see preset-based runtime model switching added.

[nicoloboschi]

you can set up an AI LLM gateway (litellm/openrouter or others) and do this config at proxy level, we're going to add an example in the next weeks

[Elliot-Construct]

but that doesn't stop the fact that some banks are different confidentiality levels and we might want one bank on a cloud model, another bank that has highly sensitive information to run through a local only model