fix(kubernetes): bind bootstrap to runtime identity

[johntmyers]

🔧 security-fix-agent

Summary

Strengthens compute-driver bootstrap authorization by correlating authenticated workloads with the durable runtime identity recorded during provisioning and restart.

Related Issue

No public issue. This change follows the private disclosure process documented in SECURITY.md.

Changes

• add an opaque runtime identity to compute-driver create, start, and authentication responses
• persist the selected driver and runtime identity with each sandbox
• require an exact durable binding match before issuing a sandbox token
• derive Kubernetes bindings from the namespace and immutable runtime resource identities
• update architecture, authentication, and operator documentation

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (local Kubernetes tooling is unavailable; CI will exercise the cluster paths)
• cargo test -p openshell-server auth::compute_driver::tests
• cargo test -p openshell-server grpc::auth_rpc::tests::issue_
• cargo test -p openshell-driver-kubernetes (228 tests)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-3531.docs.buildwithfern.com/openshell

[drew]

gator-agent

PR Review Status

This security fix is project-valid and the documentation covers the changed operator behavior, but two lifecycle gaps can leave supported sandboxes unavailable or inconsistent.

Action required: address both blocking findings and add the requested upgrade and compensation regression coverage.

Blocking findings:

• GATOR-76107d20-01: migrate or safely establish durable bindings for sandboxes created before this change.
• GATOR-76107d20-02: compensate when binding validation or persistence fails after a driver create or start succeeds.

Carried findings:

• None

Gator metadata

• Validation: Repository-admin-authored security fix with architecture and operator documentation
• Docs: Fern and architecture documentation updated
• Checks: Current-head branch checks are still running; pipeline handoff is deferred until review findings are resolved
• E2E: Kubernetes lifecycle and authentication coverage will be required after review resolution
• Head SHA: 76107d20b9faf337ba9b14002f96f00303047f9e
• Base SHA: cb6e88acb7fda83cd4355330b24dab1196105729
• Merge base SHA: 251f77e2b8f6c8d1a81236e3eea7b9f2cee6f728
• Patch ID: ebfaac05b638c0a01a7eeb4da1a0c5962970e781
• Gator payload: 10
• Review mode: initial
• Previous reviewed SHA: none
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[drew]

gator-agent

Blocked

Thanks @johntmyers. I see the new commit intended to address the lifecycle-compensation finding, and @drew has explicitly waived the backward-compatibility requirement. I cannot complete the follow-up review because this head now conflicts with main.

Next action: @johntmyers, update the branch to resolve the merge conflict. Gator will review the author-only delta and the remaining compensation obligation on the resulting head.

Gator metadata

• Head SHA: a1c846021de6d4705d0a933a8db3ba3b5fea2e70
• Gator payload: 10
• Next state: gator:blocked
• Blocked reason: merge_conflict

[drew]

gator-agent

PR Review Status

Thanks @johntmyers. I checked the rebased compensation update: the start rollback and its regression coverage now address that half of the lifecycle finding, and @drew's backward-compatibility waiver remains honored. The create path still has one reachable form of the existing compensation obligation.

Action required: make backend deletion best-effort even when the persistence failure also prevents recording the Deleting transition, and add the persistent-store-failure regression case.

Blocking findings:

• No new blocking findings.

Carried findings:

• GATOR-76107d20-02 — Create compensation can still stop before backend deletion. After the driver successfully creates a runtime, a binding-store outage can also make begin_sandbox_delete_with_initial_snapshot fail. The current path returns before calling DeleteSandbox, leaving a live, unauthenticable backend while the API reports failure. Release the global guard and attempt the idempotent backend deletion even when the durable cleanup transition cannot be stored. Verify with a store trigger that rejects all post-insert payload updates and assert that deletion is still invoked.

Gator metadata

• Validation: Repository-admin-authored security fix following the private disclosure process
• Docs: Fern and architecture documentation updated
• Checks: Current-head branch checks are still running; pipeline handoff remains deferred while the carried finding is open
• E2E: Kubernetes lifecycle and authentication coverage is required after review resolution; test workflows have not been dispatched
• Head SHA: 160b87a2b2ff233bb56180205d43e32df234c802
• Base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
• Merge base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
• Patch ID: 62ecb14cad698bfb2323f87c54d095678255555e
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: 76107d20b9faf337ba9b14002f96f00303047f9e
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:in-review

[github-actions]

Label test:e2e-kubernetes applied for b47d0c1. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA and credential-driver E2E after building the required gateway, sandbox, and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

[github-actions]

Label test:e2e applied for b47d0c1. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[drew]

gator-agent

PR Review Status

Thanks @johntmyers. I reviewed the latest cleanup commit and confirmed that backend deletion is now attempted even when the durable Deleting transition cannot be stored; the new regression test covers that failure path. The carried compensation finding is resolved, and the maintainer waiver for backward compatibility remains honored.

Action required: a maintainer must open workflow run 35666878864 and choose Re-run all jobs so the newly applied test:e2e and test:e2e-kubernetes labels actually dispatch their suites.

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-76107d20-02: resolved by the current cleanup path and regression coverage.

Gator metadata

• Validation: Repository-member-authored security fix following the private disclosure process
• Docs: Fern and architecture documentation updated
• Checks: Branch checks are running; required E2E workflows have not been dispatched
• E2E: test:e2e and test:e2e-kubernetes applied; workflow run 35666878864 requires a full rerun
• Head SHA: b47d0c1d40d1d2c09bab89312042cad0f05fce80
• Base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
• Merge base SHA: 96c08f111b7ccf3d048a876c23e724014babb585
• Patch ID: aee3a863694b45fea7a462728e2c437fe2b7ac9a
• Gator payload: 10
• Review mode: follow_up
• Previous reviewed SHA: 160b87a2b2ff233bb56180205d43e32df234c802
• Review budget exhausted: no
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[drew]

gator-agent

PR Review Status

Thanks @johntmyers. I reviewed the author-only delta after the rebase and lifecycle-race update in critical-only mode. The previously resolved compensation finding and the maintainer's backward-compatibility waiver remain intact, and no newly introduced Critical defect was found.

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-76107d20-01: waived by the maintainer; not reopened.
• GATOR-76107d20-02: remains resolved by the compensation paths and regression coverage.

Gator metadata

• Validation: Repository-member-authored security fix following the private disclosure process
• Docs: Fern, architecture, and operator documentation updated
• Checks: Current-head Branch Checks and required E2E are running; Helm Lint and Trivy Changes are green
• E2E: test:e2e and test:e2e-kubernetes are applied and the current-head workflow is dispatched
• Head SHA: dcd5f1870c34a60a088d57311d00e8cf1f659ba4
• Base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Merge base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Patch ID: 090f6eca9d8881d1c664d280b99966b840e89c0b
• Gator payload: 10
• Review mode: critical_only
• Previous reviewed SHA: b47d0c1d40d1d2c09bab89312042cad0f05fce80
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[drew]

🔒 security-review-agent

Reviewed head dcd5f1870c34a60a088d57311d00e8cf1f659ba4

This PR addresses the original UUID-only bootstrap check using the plan's durable-binding alternative: create/start return a runtime identity, the gateway persists it, and bootstrap requires an exact match. Retaining ServiceAccount authentication is a valid design choice. Two gaps remain:

1. Restart does not preserve the trusted namespace/CR binding. Kubernetes still selects the existing CR through a label-based lookup. The gateway then replaces its persisted binding with the returned identity, without supplying or checking the previously provisioned namespace/CR identity. A legitimate restart can therefore establish a binding to a different resource. Pin the original namespace and CR UID across restart, reject ambiguous matches, and permit only the expected supervisor replacement. See binding replacement and Kubernetes lookup.

2. The binding ends at token issuance. The handler still mints a legacy JWT containing no runtime binding or generation. Subsequent authentication does not consult the new persisted binding. Previously issued tokens remain usable after upgrade, and replacing a runtime does not invalidate its legacy token. Use generation-bound session credentials or enforce the binding during bearer authentication, with explicit handling of outstanding tokens. See token issuance and legacy validation.

The compensation tests, capability rejection, documentation, and skill updates are useful. Please add regression coverage for namespace/CR continuity across restart and for the actual authentication chain through provider access, including rejection of credentials from an obsolete runtime.

At the time of review, Kubernetes smoke, managed/operator workspace, and external-driver CI had passed; broader Rust checks were still running. The recorded backward-compatibility waiver is honored and is not a finding.

Validation was static code review; no local tests or live exploit reproduction were performed.

[drew]

gator-agent

PR Review Status

Thanks @drew. I checked the namespace/CR continuity and obsolete-session concerns you raised against @johntmyers's latest author-only delta. Restart now requires one resource matching the persisted namespace and CR UID, while issued supervisor credentials are generation-bound and rejected after runtime replacement. The critical-only review found no newly introduced Critical defect.

Blocking findings:

• No blocking findings remain.

Carried findings:

• GATOR-76107d20-01: remains waived by the maintainer; not reopened.
• GATOR-76107d20-02: remains resolved by the compensation paths and regression coverage.

Gator metadata

• Validation: Repository-member-authored security fix following the private disclosure process
• Docs: Fern, architecture, and operator documentation updated
• Checks: Current-head Branch Checks and required E2E are running; Helm Lint and Trivy Changes are green
• E2E: test:e2e and test:e2e-kubernetes are applied and the current-head workflow is running
• Head SHA: 6a900c60ff3ab659cf6643279b9d8369af85d182
• Base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Merge base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Patch ID: 1a98ba64fb2bb4b4e47c345ad57ddc1e722c9d4c
• Gator payload: 10
• Review mode: critical_only
• Previous reviewed SHA: dcd5f1870c34a60a088d57311d00e8cf1f659ba4
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:watch-pipeline

[drew]

gator-agent

PR Review Status

Thanks @drew. I reviewed your runtime-binding capability refactor on the latest head in critical-only mode. The earlier backward-compatibility waiver and resolved lifecycle-compensation finding remain intact, and no newly introduced Critical defect was found.

The PR cannot return to approval yet: the current-head OpenShell / E2E gate failed in rust-docker, while the Kubernetes HA and credential-driver suites required by test:e2e-kubernetes were skipped. The label-help workflow requires a full rerun before those suites are actually dispatched.

Action required: a maintainer must open workflow run 35683511049 and choose Re-run all jobs. Gator will inspect the rerun and diagnose any repeatable failure.

Blocking findings:

• No code-review blockers remain.

Carried findings:

• GATOR-76107d20-01: remains waived by the maintainer; not reopened.
• GATOR-76107d20-02: remains resolved by the compensation paths and regression coverage.

Gator metadata

• Validation: Repository-member-authored security fix following the private disclosure process
• Docs: Fern, architecture, and operator documentation updated
• Checks: Branch Checks, Helm Lint, and Trivy Changes are green; current-head OpenShell / E2E failed
• E2E: test:e2e and test:e2e-kubernetes are applied; Kubernetes HA and credential-driver jobs were skipped and require a full workflow rerun
• Head SHA: 915620967a7c2b6834aa15d216858df818ae3ec7
• Base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Merge base SHA: 99ed6a9df09a70981accfd39fe234fa1a648a93c
• Patch ID: ef6e8ebd54fae6070b6dbefea205a2347aa1803a
• Gator payload: 10
• Review mode: critical_only
• Previous reviewed SHA: 6a900c60ff3ab659cf6643279b9d8369af85d182
• Review budget exhausted: yes
• Maintainer decision required: no
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[drew]

gator-agent

Monitoring Complete

Monitoring is complete because this PR has merged.

Final status: the required Branch Checks and E2E gate succeeded before merge, and no code-review blockers remained. The Kubernetes HA and credential-driver jobs were optional and skipped.

I removed the active gator:* label because there is nothing left for gator to monitor on this PR.

Gator metadata

• Head SHA: 915620967a7c2b6834aa15d216858df818ae3ec7
• Gator payload: 10
• Final state: merged