Skip to content

fix: compare access groups as Keycloak group paths - #257

Merged
dcmcand merged 2 commits into
mainfrom
fix/232-group-paths
Nov 20, 2026
Merged

dcmcand merged 2 commits into
mainfrom
fix/232-group-paths

Conversation

@dcmcand

@dcmcand dcmcand commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Closes #232

Problem

NIC's Keycloak puts full group paths in the token's groups claim (/llm, /team-a). The pack compared the bare names in access.groups exactly, so on a NIC cluster a user in llm saw no models and couldn't create a key, the internal endpoint's group rule could never match, and the usage collector billed every request to team none.

Change

A new chart value, auth.oidc.groupsClaimForm (env LLM_OIDC_GROUPS_CLAIM_FORM), says how the groups claim writes its values:

  • path (default): Keycloak group paths. llm and /llm in access.groups both name the top-level group llm; a nested group is written as its path (/org/llm). This matches how the nebari-operator treats NebariApp groups (NormalizeGroupPaths) and how the usage console already compares groups.
  • name: values compared exactly as written, for a claim that is not a group path, such as realm_access.roles. Normalizing a role to /ops would lock everyone out, so role setups set this.

Per component:

  • Key manager: FilterModelsForUser normalizes the user's groups and the model's groups to the configured form before comparing. The normalizer is shared/oidc GroupForm.
  • Operator: the internal endpoint's SecurityPolicy allow-list holds group paths in path form (deduplicated and sorted). Name form renders values as before. An unknown form fails startup.
  • Admission webhooks: LLMModel and PassthroughModel reject a group that is blank or only slashes. Envoy Gateway refuses a JWT claim rule with no values (MinItems=1 on JWTClaim.Values), so a model whose only group normalizes to nothing would otherwise get a SecurityPolicy the API server rejects.
  • Usage collector: a team is named by its group's last path segment, so /team-a and /org/team-a bill to team-a. Bare names behave as before.
  • Keycloak mappers: the pack's own group mapper on the key manager's SPA client and the dev realm now set full.path: "true", as NIC's realm does.
  • Docs: configuration.mdx documents the new value; the installation runbook and troubleshooting page explain the path form and nested groups.

Two existing expectations change with the new default: operator/.../auth_test.go now expects the allow-list [/ml-team /platform], and dev/usage-forgery-check.sh expects the recorded groups to include /team-a.

Testing

  • shared, key-manager, usage-console/server: go test ./... pass.
  • operator: make test (envtest) passes.
  • Usage collector: make test-usage-collector passes, with two new cases for path groups (API-key owner and a nested group in the verified token).
  • Mutation checks: removing the key manager's normalization, the operator's path conversion, or the webhook's group-name check each turns a test red.
  • helm lint passes; helm template renders LLM_OIDC_GROUPS_CLAIM_FORM for the operator and key manager, and full.path: "true" on the SPA mapper.

Dev end-to-end on a kind cluster (make setup), with the dev realm now issuing group paths (alice's token carries ["/llm-users","/team-a"]):

  • dev/usage-forgery-check.sh: all checks pass. Alice's token gets 200 on the internal endpoint; a valid token with no groups gets 403 (rbac_access_denied_matched_policy[DENY]). The rendered SecurityPolicy for PassthroughModel mock-provider (access.groups: ["llm","llm-users"]) holds the claim values /llm and /llm-users.
  • dev/usage-e2e.sh: passes. Alice and bob sign in with path-form tokens, see the model and create keys; usage is billed to team-a (11 requests) and team-b (7) in OpenMeter and the usage console.
  • make test-auth: valid token 200, missing 401, garbage 401. The script also prints 0: arithmetic syntax error in expression, which comes from the script and does not affect the result.
  • make test-model-scope: all cases pass.

Not run yet: a deploy on a NIC cluster, where NIC's realm-level groups mapper sits alongside the pack's client mapper.

Notes

  • A model already stored with a slash-only group is not re-checked by the webhook; its SecurityPolicy would now be rejected where before it matched nothing.
  • The path normalizer exists in three places: shared/oidc, the operator (its image builds from operator/ alone and cannot import shared), and the usage console's existing groupPath.

@github-actions

Copy link
Copy Markdown

📄 Docs preview for fix/232-group-paths:
https://fix-232-group-paths.llm-serving-pack.pages.dev

@dcmcand
dcmcand requested review from aktech and jbouder November 20, 2026 10:48

@jbouder jbouder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One blocker: the new path default needs an upgrade note. Two smaller fixes inline.

Comment thread charts/nebari-llm-serving/values.yaml Resolved
Comment thread key-manager/internal/models/watcher.go Resolved
@dcmcand
dcmcand force-pushed the fix/232-group-paths branch from eb9a4ee to c988069 Compare November 20, 2026 13:35
@dcmcand
dcmcand requested a review from jbouder November 20, 2026 13:55
NIC's Keycloak puts full group paths in the groups claim (/llm), while the pack compared the bare names in access.groups exactly, so a NIC user in llm saw no models, the internal endpoint's group rule could not match, and usage was billed to team none.

A new chart value, auth.oidc.groupsClaimForm (LLM_OIDC_GROUPS_CLAIM_FORM), says how the claim writes its values. In path form, the default, the key manager compares both sides as paths and the operator writes the gateway's allow-list as paths, so llm and /llm both name the top-level group; in name form values are compared as written, for a claim such as realm_access.roles. The admission webhooks reject a group that is blank or only slashes, since Envoy Gateway refuses a claim rule with no values. The usage collector names a team by its group's last path segment. The pack's own group mapper and the dev realm now emit paths, as NIC's realm does.

Closes #232
The auditor compared a model's access groups with the userinfo groups
exactly, so in path form a model's "llm" never matched the "/llm" Keycloak
returns and a user still in the group would lose their key. NewAuditor now
takes the group form and normalizes both sides, as the model watcher does.

The key manager's NebariApp mapper hardcoded full.path "true", so with
groupsClaimForm: name tokens still carried "/llm" and private models
vanished from the key manager. full.path now follows groupsClaimForm.

Add an upgrading entry for installs whose tokens carry plain group names:
role claims set groupsClaimForm: name, and group mappers with full path
off turn it on.
@dcmcand
dcmcand force-pushed the fix/232-group-paths branch from c988069 to 9ff27e6 Compare November 20, 2026 13:56

@jbouder jbouder left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three addressed. LGTM.

@dcmcand
dcmcand merged commit d5f05bf into main Nov 20, 2026
21 checks passed
@dcmcand
dcmcand deleted the fix/232-group-paths branch November 20, 2026 16:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Group checks fail on NIC: Keycloak sends full group paths and the pack compares bare names

2 participants