Support for LocalAI provider

[eode]

I'm using an OpenAI-compatible backend (LocalAI) and Juggler assumes the context of all models is 8k. That's... ..pretty slim. It can edit each model manually, but it would be nice to have automatic context size recognition.

Note: If you at all feel like adding LocalAI as a provider, it's OpenAI-compatible plus some extras, and the call to retrieve context length is:

# Post to /api/models/vram-estimate
{
  "context_size": 0,
  "gpu_layers": 0,
  "kv_quant_bits": 0,
  "model": "<model string>"
}

..technically a call for to estimate vram usage, but its return value includes the max context length that is currently configured for that model.

Edit summary: Changed title, removed nonissue

[julianstorer]

Hello there

The 8k is the llama.cpp provider's fallback, which kicks in when it can't find out the real size. Since it defaults to the same port LocalAI uses, pointing it there works for chat but leaves it guessing, because LocalAI's model list doesn't mention context size and the other places we look don't exist on it.

Worth knowing that 8k is often genuinely right, since LocalAI caps the context at 8k itself unless the model's config says otherwise. We'd rather not use the VRAM endpoint you mentioned: it needs admin access, disappears on servers running without the web UI, and the size it reports is the model's theoretical maximum rather than what the server is actually serving — so we'd send far more than it can take and have it quietly dropped. There's a simpler endpoint (/api/show) that needs no login and reports the size actually in use, which looks like the way to do this.

Which provider did you pick in settings, out of interest?

[eode]

Ah. /api/show isn't listed in the swagger API on my instance, but it seems to work. Glad there's an alternative.

8k is a value that is often configured away from. Yes, it's a default, but it's not exactly the most commonly used setting. Heck, a lot of agents will use more than half of that just loading up.

I tried ollama, but LocalAI returns modelname<:tag>, i.e., modelname:latest, but LocalAI doesn't accept a model request for modelname:latest - the :latest must be stripped first.

I tried OpenAI-compatible custom endpoint, as well.

[julianstorer]

Thanks

I think I sent you down the wrong path. /api/show answers, but LocalAI hardcodes its parameters field empty, and that's the field we'd have parsed. The context length only shows up under model_info["general.context_length"], and only for models with an explicit context_size.

The one to use is GET /v1/models/capabilities — no auth, one call for every model, and its context_size is the backend's effective context rather than a theoretical maximum.

So: LocalAI gets its own provider, listing from /v1/models and taking windows from capabilities. Custom OpenAI-compatible endpoints get the same probe, so they stop assuming 128k for anything they can't identify. I'll run an instance here and build against real responses before any of it lands.

On :latest — current LocalAI strips that suffix in middleware before looking the model up. Which version are you on? If it's recent and still rejecting, that's a separate bug and I'd want the error text.

And fair hit on 8k.

[julianstorer]

OK.. I've pushed some work on this to the develop branch. Will be in the next release build

LocalAI is now a provider of its own — it lists from /v1/models and takes each model's window from /v1/models/capabilities, so the number comes from your server rather than a guess, and custom OpenAI-compatible endpoints get the same probe. Two things worth checking your end: only a top-level context_size: changes what LocalAI serves (under parameters:, where the gallery configs put it, it's ignored, so those models really are 8192), and the window is only readable on v4.10.0 or newer. On :latest, v4.10.0 accepted both qwen3-0.6b and qwen3-0.6b:latest here, so that looks fixed upstream — if you're running the latest-cpu image, note it's still v3.0.0 from last year, which would explain both symptoms.

[eode]

Thanks! I'll give it a try once that comes out, and report back my findings here. I'm on 4.10.0, as of some time within the last few days. Currently on an ROCM image, but was previously on Vulkan.

In any case, the behavior of /v1/models/capabilities is.. ..well, decent on my system. Not perfect, but good enough.

• My global default is set to 16384. If a model is not set, v1/models/capabilities ignores the global default and states 4096 (not on you, obviously). Edit: bug report opened with LocalAI
• context_size in model's yml is presented if set.

Good to know that they've fixed the ":latest" issue.

I'll be keeping an eye out for the next Juggler.