feat(sandbox): add delegated identity for token exchange

[grs]

Summary

Add opt-in delegated OIDC identity support for sandboxes so token-exchange provider profiles can use the sandbox creator’s bounded identity instead of a stored provider subject credential. This adds gateway managed delegated credential storage, sandbox delegation lifecycle APIs, CLI commands for enabling, extending, withdrawing, and administering delegation.

Note this builds on PR #1970, which adds the basic token exchange capability.

Example CLI flow:

openshell sandbox create \
  --provider protected-services \
  --delegate-identity-for=8h

openshell sandbox delegated-identity status my-sandbox
openshell sandbox delegated-identity extend my-sandbox --for=12h
openshell sandbox delegated-identity withdraw my-sandbox

Related Issue

Fixes #4599

Changes

• Added sandbox_delegated_identity as a token-exchange subject source.
• Added sandbox delegation status, withdraw, and extend APIs.
• Added platform-admin delegated credential list, status, revoke, and delete APIs.
• Added openshell sandbox create --delegate-identity-for and related CLI commands.
• Updated provider docs, sandbox architecture docs, tests, and generated Go proto bindings.

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)
• Manual testing with different users/roles

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[zanetworker]

The --delegate-identity-for flag name makes this feel like infrastructure plumbing when it's really a trust decision. Something like --act-as-me (or something like that) immediately communicates what the user is agreeing to without requiring identity semantics knowledge.

# Current
openshell sandbox create --provider svc --delegate-identity-for=8h

# Options
openshell sandbox create --provider svc --act-as-me-for=8h
openshell sandbox create --provider svc --act-on-my-behalf-for=8h
...

[grs]

The --delegate-identity-for flag name makes this feel like infrastructure plumbing when it's really a trust decision. Something like --act-as-me (or something like that) immediately communicates what the user is agreeing to without requiring identity semantics knowledge.

@zanetworker while I understand that --act-as-me is more free of technical jargon, my concern is that the mental model it suggests is of general impersonation, while the implementation is the much narrower scoped identity delegation for token exchange (assuming that is configured separately in a provider profile in use).

Would --use-my-identity-for address your concerns? To me that feels a bit closer to the scope than --act-as-me, and could further be clarified in the help text for the flag.

[politerealism]

Chiming in on the flag naming — "on-behalf-of" might be worth considering: --on-behalf-of-for=8h (or --on-behalf-of=8h, since it's always the creator's own identity being delegated). It's the established OAuth/enterprise-identity term for exactly this pattern (Microsoft's OBO flow, etc.), so it reads as precise and scoped to people who know the space, without implying the broader "full impersonation" connotation --act-as-me carries, or being as opaque as --delegate-identity-for/--use-my-identity-for about what's actually happening.

That said, I don't think the flag name alone can fully carry the "this is a trust decision" signal either way. Might be worth pairing whichever name wins with an interactive confirmation at creation time — something like "This lets <provider> present tokens on your behalf for 8h — continue? [y/N]" — so the caution is delivered where it actually lands, rather than trying to pack it all into the flag itself.

[grs]

Chiming in on the flag naming — "on-behalf-of" might be worth considering: --on-behalf-of-for=8h (or --on-behalf-of=8h, since it's always the creator's own identity being delegated).

I do like that. It does feel as though the value should be who it's acting on behalf of though. --on-behalf-of-me-for would be clearer but perhaps a bit too long, or --on-my-behalf-for=8h maybe?

[lbelyaev]

Orthogonal to the flag-naming thread, but on the same "this is a trust decision" theme — two questions on the withdrawal semantics, which are new here (the exchange flow in #1970 and the design in #1987 don't cover a delegation lifecycle). Both fit the least-privilege posture #1987 already commits to, so flagging where the current behavior loosens it.

First, withdraw gates the next resolve (resolve_subject_access_token checks withdrawn_at_ms before exchanging), but a token already exchanged and held by the sandbox keeps its own access_token_expires_at_ms — so withdraw means "mint no more," not "access stops now." For an 8h delegation the effective revocation lag is the outstanding token's lifetime. Is lazy-on-next-resolve the intended contract, or should withdraw also bound the outstanding token (a short exchanged-token TTL, or a downstream revoke)? There's a separate revoke for the credential, but a delegator reaching for withdraw may reasonably expect in-flight access to stop too.

Second, handle_extend resets withdrawn_at_ms = 0, so extend both prolongs and un-withdraws — a withdrawn delegation is revived by an extend call with no separate re-authorization. If withdraw is meant to read as a revocation act, should reviving a withdrawn delegation be an explicit re-delegate rather than an extend silently clearing the flag?

[grs]

Orthogonal to the flag-naming thread, but on the same "this is a trust decision" theme — two questions on the withdrawal semantics, which are new here (the exchange flow in #1970 and the design in #1987 don't cover a delegation lifecycle). Both fit the least-privilege posture #1987 already commits to, so flagging where the current behavior loosens it.

First, withdraw gates the next resolve (resolve_subject_access_token checks withdrawn_at_ms before exchanging), but a token already exchanged and held by the sandbox keeps its own access_token_expires_at_ms — so withdraw means "mint no more," not "access stops now." For an 8h delegation the effective revocation lag is the outstanding token's lifetime. Is lazy-on-next-resolve the intended contract, or should withdraw also bound the outstanding token (a short exchanged-token TTL, or a downstream revoke)? There's a separate revoke for the credential, but a delegator reaching for withdraw may reasonably expect in-flight access to stop too.

It is deliberate, mainly to keep things simple without requiring a check on every invocation. However the max time a token can be cached for can be limited through the config independently of the actual expiration on the issued token, so the 'lag' can be restricted. If you allow a token to be cached for a max of 2 minutes, say, you still get the benefit of caching and not having to check the validity on every relevant request, but you don't need to wait long before you know there will be no more requests. (If instant termination of requests is desired, probably killing the sandbox would be the first lever).

Second, handle_extend resets withdrawn_at_ms = 0, so extend both prolongs and un-withdraws — a withdrawn delegation is revived by an extend call with no separate re-authorization. If withdraw is meant to read as a revocation act, should reviving a withdrawn delegation be an explicit re-delegate rather than an extend silently clearing the flag?

That was again intentional. Withdraw is not a revocation, just an 'undo' of a previous extend. You can extend if the previous delegation has expired. Redelegating after a withdraw seems possibly desirable and it didn't make a lot of sense to me to just have another verb for that. If the admin revokes, the user cannot extend until the admin deletes the record.

Does this make sense?

[lbelyaev]

Orthogonal to the flag-naming thread, but on the same "this is a trust decision" theme — two questions on the withdrawal semantics, which are new here (the exchange flow in #1970 and the design in #1987 don't cover a delegation lifecycle). Both fit the least-privilege posture #1987 already commits to, so flagging where the current behavior loosens it.
First, withdraw gates the next resolve (resolve_subject_access_token checks withdrawn_at_ms before exchanging), but a token already exchanged and held by the sandbox keeps its own access_token_expires_at_ms — so withdraw means "mint no more," not "access stops now." For an 8h delegation the effective revocation lag is the outstanding token's lifetime. Is lazy-on-next-resolve the intended contract, or should withdraw also bound the outstanding token (a short exchanged-token TTL, or a downstream revoke)? There's a separate revoke for the credential, but a delegator reaching for withdraw may reasonably expect in-flight access to stop too.

It is deliberate, mainly to keep things simple without requiring a check on every invocation. However the max time a token can be cached for can be limited through the config independently of the actual expiration on the issued token, so the 'lag' can be restricted. If you allow a token to be cached for a max of 2 minutes, say, you still get the benefit of caching and not having to check the validity on every relevant request, but you don't need to wait long before you know there will be no more requests. (If instant termination of requests is desired, probably killing the sandbox would be the first lever).

Second, handle_extend resets withdrawn_at_ms = 0, so extend both prolongs and un-withdraws — a withdrawn delegation is revived by an extend call with no separate re-authorization. If withdraw is meant to read as a revocation act, should reviving a withdrawn delegation be an explicit re-delegate rather than an extend silently clearing the flag?

That was again intentional. Withdraw is not a revocation, just an 'undo' of a previous extend. You can extend if the previous delegation has expired. Redelegating after a withdraw seems possibly desirable and it didn't make a lot of sense to me to just have another verb for that. If the admin revokes, the user cannot extend until the admin deletes the record.

Does this make sense?

Makes sense on both — thanks. The configurable cache cap was the piece I was missing; that bounds the lag independently of the token's own lifetime, and killing the sandbox covers the instant-stop case. And framing withdraw as the undo of extend, with revoke as the separate admin gate, is a clean split — I'd conflated the two. Appreciate the details.

[zanetworker]

@grs given the long discussion, I am good with "--use-my-identity-for" as an improvement from delegate-identity-for as its clear on which identity is being consumed here (the user's).

[grs]

@grs given the long discussion, I am good with "--use-my-identity-for" as an improvement from delegate-identity-for as its clear on which identity is being consumed here (the user's).

I've made that change.

[mrunalp]

Findings

1. Workspace admins can redirect another user’s OIDC token. A workspace admin may update an attached custom profile’s token_endpoint; exchanges reload the mutable profile and send the delegator’s raw subject token to that endpoint. This bypasses the delegated-sandbox ownership
   boundary. Bind security-sensitive profile fields at delegation time or prohibit such mutations while delegated sandboxes reference the profile. provider.rs:2767
   (

   OpenShell/crates/openshell-server/src/grpc/provider.rs

   Line 2767 in b0a30dd

   pub(super) async fn handle_update_provider_profiles(

   ), provider.rs:3818
   (

   OpenShell/crates/openshell-server/src/grpc/provider.rs

   Line 3818 in b0a30dd

   let catalog = state

   ), provider.rs:3924
   (

   OpenShell/crates/openshell-server/src/grpc/provider.rs

   Line 3924 in b0a30dd

   let token_response = perform_intermediate_token_exchange(

   )

2. Expiry or withdrawal is not rechecked after refresh. The gateway validates the delegation, then may wait for a lease and perform several network/storage operations before returning the token. A withdrawal or expiry during that interval can still result in a newly minted and
   cached token afterward. delegated_identity.rs:177 (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 177 in b0a30dd

   let record = sandbox_delegated_identity_record(state, sandbox).await?;

   ), delegated_identity.rs:203
   (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 203 in b0a30dd

   let resolved = refresh_if_needed(state, credential).await?;

   )

3. The refresh lease can expire while its owner is still refreshing. The fixed 120-second lease is not renewed, while discovery, refresh, JWT validation, credential storage, and persistence have no combined deadline. In HA, another replica can take over and replay a rotating
   refresh token, potentially consuming it and forcing reauthorization. delegated_identity.rs:51 (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 51 in b0a30dd

   const REFRESH_LEASE_DURATION_MS: i64 = 120_000;

   ), delegated_identity.rs:1501
   (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 1501 in b0a30dd

   async fn refresh_with_owned_lease_inner(

   )

4. Cleanup CAS conflicts can permanently orphan OAuth material. Failed staged-secret deletion records its handles using one read/CAS attempt. A concurrent update causes the handles to be logged and discarded, leaving tokens permanently in the credential backend.
   delegated_identity.rs:1227 (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 1227 in b0a30dd

   if let Err(error) = state

   ), delegated_identity.rs:1243
   (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 1243 in b0a30dd

   async fn persist_staged_cleanup_handles(

   )

5. Validation and runtime disagree on whitespace. source: " sandbox_delegated_identity " passes profile validation because it is trimmed, but runtime exact-matching rejects it as unsupported. Normalize during lowering/runtime or reject noncanonical input. profiles.rs:2828
   (

   OpenShell/crates/openshell-providers/src/profiles.rs

   Line 2828 in b0a30dd

   let subject_credential = subject_token.credential.trim();

   ), provider.rs:3851
   (

   OpenShell/crates/openshell-server/src/grpc/provider.rs

   Line 3851 in b0a30dd

   let (stored_subject_token, subject_token_expires_at_ms, subject_cache_key) =

   )

6. --for durations depend on the CLI clock. The CLI converts durations to absolute timestamps using local time, while the gateway validates against its clock. Clock skew changes the requested duration and can make the documented maximum fail. Use gateway now_ms or send a
   duration. run.rs:1515 (

   OpenShell/crates/openshell-cli/src/run.rs

   Line 1515 in b0a30dd

   fn delegated_identity_request(duration_ms: i64) -> DelegatedIdentityRequest {

   ), delegated_identity.rs:890
   (

   OpenShell/crates/openshell-server/src/delegated_identity.rs

   Line 890 in b0a30dd

   fn validate_delegation_window(

   )

Design notes

• Delegated OAuth material is separated from the normal CLI login and stored behind credential-driver handles.
• Sandbox interaction is restricted to the delegator; admins retain stop/delete containment controls.
• Admin revocation is intentionally recoverable through user reauthorization—it is not a durable administrative ban. That terminology deserves explicit sign-off.

[grs]

Findings

1. Workspace admins can redirect another user’s OIDC token. A workspace admin may update an attached custom profile’s token_endpoint; exchanges reload the mutable profile and send the delegator’s raw subject token to that endpoint. This bypasses the delegated-sandbox ownership boundary.

In my view this is acceptable limitation of the current authorization model. Workspace administrators control provider exchange configuration and can already redirect provider-credential subject tokens. Delegated identity adds per-user sandbox access restrictions, but does not protect subject access tokens from administrators controlling an attached custom profile. I have updated the documentation to clarify this.

2. Expiry or withdrawal is not rechecked after refresh. The gateway validates the delegation, then may wait for a lease and perform several network/storage operations before returning the token. A withdrawal or expiry during that interval can still result in a newly minted and cached token afterward.

Again, I think it is acceptable that withdrawal of delegated identity is not instantaneous but the lag is bounded. This is also true for cached tokens where trying to ensure immediate action on withdrawal would either prevent caching or add complexity. Where drastic action is required, the sandbox can always be deleted. I have updated the documentation.

3. The refresh lease can expire while its owner is still refreshing. The fixed 120-second lease is not renewed, while discovery, refresh, JWT validation, credential storage, and persistence have no combined deadline. In HA, another replica can take over and replay a rotating refresh token, potentially consuming it and forcing reauthorization.

Fixed

4. Cleanup CAS conflicts can permanently orphan OAuth material. Failed staged-secret deletion records its handles using one read/CAS attempt. A concurrent update causes the handles to be logged and discarded, leaving tokens permanently in the credential backend.

Fixed

5. Validation and runtime disagree on whitespace. source: " sandbox_delegated_identity " passes profile validation because it is trimmed, but runtime exact-matching rejects it as unsupported. Normalize during lowering/runtime or reject noncanonical input.

Fixed

6. --for durations depend on the CLI clock. The CLI converts durations to absolute timestamps using local time, while the gateway validates against its clock. Clock skew changes the requested duration and can make the documented maximum fail. Use gateway now_ms or send a

Fixed

[grs]

@mrunalp Thank you for review! I've fixed most issues and commented on the other two above.

[mrunalp]

Findings

• Cancelled sandbox creation can orphan an active delegation. The delegation is persisted before compute creation, while cleanup occurs only when creation returns an error. Cancellation/crash can skip cleanup, and credential deletion then treats the orphan as active until expiry.
  sandbox.rs (

  OpenShell/crates/openshell-server/src/grpc/sandbox.rs

  Lines 539 to 565 in f038eaf

  	crate::delegated_identity::store_prepared_sandbox_delegation(
  	state,
  	delegated_identity.as_ref(),
  	)
  	.await?;
  	let mut sandbox = match state
  	.compute
  	.create_sandbox(sandbox, sandbox_token, await_main_process_attachment)
  	.await
  	{
  	Ok(sandbox) => sandbox,
  	Err(status) => {
  	if let Err(cleanup_status) =
  	crate::delegated_identity::delete_prepared_sandbox_delegation(
  	state,
  	delegated_identity.as_ref(),
  	)
  	.await
  	{
  	warn!(
  	sandbox_id = %id,
  	error = %cleanup_status,
  	"failed to clean up prepared delegated identity after sandbox create failure"
  	);
  	}
  	return Err(status);
  	}

  ), delegated_identity.rs
  (

  OpenShell/crates/openshell-server/src/delegated_identity.rs

  Lines 713 to 740 in f038eaf

  	async fn ensure_no_active_delegation_references(
  	state: &Arc<ServerState>,
  	credential_id: &str,
  	) -> Result<(), Status> {
  	let now = current_time_ms();
  	let records = state
  	.store
  	.collect_messages::<SandboxDelegatedIdentityRecord>(ObjectListQuery::AllWorkspaces)
  	.await
  	.map_err(|e| {
  	Status::internal(format!(
  	"list sandbox delegated identity references failed: {e}"
  	))
  	})?;
  	if records.iter().any(|record| {
  	record
  	.delegated_identity
  	.as_ref()
  	.is_some_and(|delegation| {
  	delegation.credential_id == credential_id
  	&& delegation.withdrawn_at_ms == 0
  	&& delegation.delegated_until_ms > now
  	})
  	}) {
  	return Err(Status::failed_precondition(
  	"delegated credential is referenced by an active sandbox delegation",
  	));
  	}

  )

• Partial secret storage can permanently orphan OAuth material. Access and refresh tokens are stored concurrently; if one write fails and cleanup also fails, the successful handles are logged then discarded, leaving no durable retry path. credentials.rs
  (

  OpenShell/crates/openshell-server/src/credentials.rs

  Lines 330 to 405 in f038eaf

  	let futures = credentials.iter().map(|(credential_key, value)| {
  	let driver_name = driver_name.clone();
  	let driver = driver.clone();
  	async move {
  	let existing_handle = existing_handles
  	.get(credential_key)
  	.filter(|handle| normalize_driver_name(&handle.driver) == driver_name)
  	.cloned();
  	let replaced_handle = existing_handles
  	.get(credential_key)
  	.filter(|handle| normalize_driver_name(&handle.driver) != driver_name)
  	.cloned();
  	let mut handle = driver
  	.store_credential(StoreCredentialRequest {
  	provider_name: provider_name.to_string(),
  	credential_key: credential_key.clone(),
  	value: value.clone(),
  	existing_handle,
  	workspace: workspace.to_string(),
  	provider_id: provider_id.to_string(),
  	object_id: object_id.to_string(),
  	})
  	.await?;
  	handle.driver.clone_from(&driver_name);
  	if handle.handle.trim().is_empty() {
  	return Err(Status::internal(format!(
  	"credential driver '{driver_name}' returned an empty handle for provider credential '{credential_key}'"
  	)));
  	}
  	if let Some(replaced_handle) = replaced_handle {
  	self.delete_provider_credential_handle(
  	provider_name,
  	workspace,
  	provider_id,
  	credential_key,
  	replaced_handle,
  	)
  	.await?;
  	}
  	Ok::<_, Status>((credential_key.clone(), handle))
  	}
  	});
  	let results = futures::future::join_all(futures).await;
  	let mut successes = HashMap::new();
  	let mut first_error: Option<Status> = None;
  	for result in results {
  	match result {
  	Ok((key, handle)) => {
  	successes.insert(key, handle);
  	}
  	Err(err) if first_error.is_none() => {
  	first_error = Some(err);
  	}
  	Err(_) => {}
  	}
  	}
  	if let Some(err) = first_error {
  	if !successes.is_empty()
  	&& let Err(cleanup_err) = self
  	.delete_provider_credential_handles(
  	provider_name,
  	workspace,
  	provider_id,
  	&successes,
  	)
  	.await
  	{
  	tracing::warn!(
  	provider_name = %provider_name,
  	error = %cleanup_err,
  	"failed to clean up partially stored credentials after error"
  	);
  	}
  	return Err(err);

  )

• Two prior security concerns were documented, not remediated. Workspace admins can still redirect a delegated subject token by changing the current profile’s token_endpoint; withdrawal/expiry still does not recheck after an in-flight refresh. The new documentation accurately
  discloses both limitations, so they now require explicit maintainer acceptance of that trust/containment model. provider.rs (

  OpenShell/crates/openshell-server/src/grpc/provider.rs

  Lines 3816 to 3937 in f038eaf

  	let provider = state
  	.store
  	.get_message_by_name::<Provider>(&workspace, &req.provider)
  	.await
  	.map_err(|e| Status::internal(format!("fetch provider failed: {e}")))?
  	.ok_or_else(|| Status::not_found("provider not found"))?;
  	let catalog = state
  	.provider_profile_sources
  	.snapshot_catalog(state.store.as_ref(), &workspace)
  	.await?;
  	let profile = get_provider_type_profile_for_scope(
  	&catalog,
  	&provider.r#type,
  	&provider.profile_workspace,
  	)
  	.ok_or_else(|| Status::failed_precondition("provider profile not found"))?;
  	let profile_proto = profile.to_proto();
  	let credential = profile_proto
  	.credentials
  	.iter()
  	.find(|credential| credential.name == req.credential_key)
  	.ok_or_else(|| {
  	Status::failed_precondition("credential not declared by provider profile")
  	})?;
  	let token_grant = credential
  	.token_grant
  	.as_ref()
  	.ok_or_else(|| Status::failed_precondition("credential does not declare token_grant"))?;
  	let grant_type = ProviderCredentialTokenGrantType::try_from(token_grant.grant_type)
  	.unwrap_or(ProviderCredentialTokenGrantType::ClientCredentials);
  	if grant_type != ProviderCredentialTokenGrantType::TokenExchange {
  	return Err(Status::failed_precondition(
  	"credential token_grant is not token_exchange",
  	));
  	}
  	let subject_token = token_grant
  	.subject_token
  	.as_ref()
  	.ok_or_else(|| Status::failed_precondition("token_exchange subject_token is missing"))?;
  	let (stored_subject_token, subject_token_expires_at_ms, subject_cache_key) =
  	match subject_token.source.as_str() {
  	"provider_credential" => {
  	if !profile_proto
  	.credentials
  	.iter()
  	.any(|credential| credential.name == subject_token.credential)
  	{
  	return Err(Status::failed_precondition(
  	"subject token credential not declared by provider profile",
  	));
  	}
  	(
  	resolve_subject_token_credential(
  	&state.credentials,
  	&provider,
  	&subject_token.credential,
  	)
  	.await?,
  	provider_credential_expires_at_ms(&provider, &subject_token.credential),
  	subject_token.credential.clone(),
  	)
  	}
  	"sandbox_delegated_identity" => {
  	if !subject_token.credential.trim().is_empty() {
  	return Err(Status::failed_precondition(
  	"sandbox_delegated_identity subject_token must not set credential",
  	));
  	}
  	let (access_token, expires_at_ms, credential_id) =
  	crate::delegated_identity::resolve_subject_access_token(state, &sandbox)
  	.await?;
  	(access_token, expires_at_ms, credential_id)
  	}
  	_ => {
  	return Err(Status::failed_precondition(
  	"unsupported subject_token source",
  	));
  	}
  	};
  	let jwt_svid_audience =
  	effective_jwt_svid_audience(&token_grant.token_endpoint, &token_grant.jwt_svid_audience);
  	let gateway_jwt_svid = fetch_gateway_jwt_svid(&jwt_svid_audience).await?;
  	let gateway_claims = parse_unverified_spiffe_claims(&gateway_jwt_svid)?;
  	validate_gateway_jwt_svid_claims(&gateway_claims, &jwt_svid_audience)?;
  	let supervisor_claims = validate_supervisor_jwt_svid(
  	&req.supervisor_jwt_svid,
  	&gateway_claims,
  	&jwt_svid_audience,
  	)
  	.await?;
  	let intermediate_cache_key = intermediate_token_cache_key(IntermediateTokenCacheKeyInput {
  	provider: &provider,
  	dynamic_credential: &req.credential_key,
  	subject_credential: &subject_cache_key,
  	token_endpoint: &token_grant.token_endpoint,
  	client_assertion_type: effective_client_assertion_type(&token_grant.client_assertion_type),
  	subject_token_type: effective_token_type(&subject_token.subject_token_type),
  	audience: &supervisor_claims.sub,
  	requested_token_type: effective_token_type(&token_grant.requested_token_type),
  	supervisor_subject: &supervisor_claims.sub,
  	gateway_subject: &gateway_claims.sub,
  	});
  	if let Some(cached) = INTERMEDIATE_TOKEN_CACHE.get(&intermediate_cache_key) {
  	return Ok(Response::new(ExchangeProviderSubjectTokenResponse {
  	access_token: cached.access_token,
  	expires_in: cached.expires_in,
  	token_type: cached.token_type,
  	}));
  	}
  	let token_response = perform_intermediate_token_exchange(
  	&token_grant.token_endpoint,
  	&gateway_jwt_svid,
  	&token_grant.client_assertion_type,
  	&stored_subject_token,
  	&subject_token.subject_token_type,
  	&supervisor_claims.sub,
  	&token_grant.requested_token_type,
  	)
  	.await

  ),
  delegated_identity.rs (

  OpenShell/crates/openshell-server/src/delegated_identity.rs

  Lines 176 to 212 in f038eaf

  	pub async fn resolve_subject_access_token(
  	state: &Arc<ServerState>,
  	sandbox: &Sandbox,
  	) -> Result<(String, i64, String), Status> {
  	let record = sandbox_delegated_identity_record(state, sandbox).await?;
  	let delegation = record
  	.as_ref()
  	.and_then(|record| record.delegated_identity.as_ref())
  	.ok_or_else(|| {
  	Status::failed_precondition("sandbox was not created with delegated identity")
  	})?;
  	if delegation.withdrawn_at_ms > 0 {
  	return Err(Status::failed_precondition("delegated identity withdrawn"));
  	}
  	let now = current_time_ms();
  	if delegation.delegated_until_ms <= now {
  	return Err(Status::failed_precondition("delegated identity expired"));
  	}
  	let credential = state
  	.store
  	.get_message::<DelegatedIdentityCredential>(&delegation.credential_id)
  	.await
  	.map_err(|e| Status::internal(format!("fetch delegated credential failed: {e}")))?
  	.ok_or_else(|| Status::failed_precondition("delegated credential missing"))?;
  	if credential.principal_subject != delegation.principal_subject {
  	return Err(Status::failed_precondition(
  	"delegated credential principal does not match sandbox delegation",
  	));
  	}
  	ensure_delegated_credential_usable(&credential)?;
  	let resolved = refresh_if_needed(state, credential).await?;
  	let credential_id = resolved.credential.object_id().to_string();
  	Ok((
  	resolved.access_token,
  	resolved.credential.access_token_expires_at_ms,
  	credential_id,
  	))

  )

Fixed since the previous review

Gateway-clock durations, lease renewal/fencing, cleanup CAS retries, profile whitespace normalization, and issue #1987’s acceptance state are resolved.

[github-actions]

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

[zanetworker]

We are running token exchange in a deployment where a sandbox agent calls an internal API as the user who owns the sandbox, and this PR covers the gap we hit. Adding the use case and two questions so it doesn't get closed as stale.

Setup. A provider profile with a token_exchange grant and subject_token.source: provider_credential. Keycloak is the IdP. Built from main at e7fdd6b; the subject-token path is unchanged on current main.

What breaks today. The stored subject token is a snapshot. --from-oidc-token saves users from pasting a token, but per the provider docs the CLI copies only the access token and its expiry, and OpenShell does not store a refresh token for it. Once that token expires, ensure_subject_token_credential_not_expired in crates/openshell-server/src/grpc/provider.rs rejects the intermediate exchange, and every request from the sandbox returns HTTP 502 until the user runs provider update again. Profile validation also blocks attaching a refresh block to a broker-only subject credential (credential_has_workload_injection_metadata in crates/openshell-providers/src/profiles.rs). That means a sandbox running for days fails within one access-token lifetime, and the user can't tell what went wrong.

sandbox_delegated_identity fixes this. The gateway holds its own refresh-token chain, scoped to a delegation window per sandbox, so the user never handles a token and long-running sandboxes keep working. We would use it as soon as it lands.

Question 1: subject token audience. With Keycloak token exchange, the gateway's intermediate exchange authenticates with the gateway JWT-SVID, so the subject token's aud has to include the client that client assertion identifies. A normal gateway login token has aud set to the gateway API client, so the exchange fails. We work around this with a separate Keycloak client and a subject token minted for that audience. The PR docs say the delegated authorization requests openid, offline_access, plus the gateway audience. Is the expectation that operators add a Keycloak audience mapper so delegated tokens also carry the gateway's token-exchange client identity? If so, documenting that IdP requirement next to the Authorization Code and offline_access requirements would save others the debugging. If that isn't enough for some IdPs, could the delegated authorization take a configurable extra audience or resource?

Question 2: failure surfacing. When the delegated grant is revoked or the refresh fails, the sandbox still gets an opaque 502. Linking this to #3319 (typed token-grant failures) would let a trusted caller see provider_authorization_required and tell the user to run delegated-identity extend.

Happy to test a rebased branch on Kubernetes with Keycloak. The docs moved under docs/how-it-works/providers/ on main since this PR was opened.

[grs]

@mrunalp I believe I've addressed the remaining issues from yourreview.

@zanetworker Yes, client access tokens should have the openshell-gateway added to them. (In my dev setup I add that as a default scope on the openshell-cli client). Once the typed errors PR is merged we can raise specific changes to what is reported as separate issues.

[EmilienM]

Hi @grs, heads up: #4147 just merged and removed sandbox_sync_guard(), so handle_extend and handle_delete_credential will need a MutationScope on rebase (see compute/mutation_guard.rs):

• Extend maps to MutationScope::sandbox(workspace, id).
• Delete-credential is the real choice. Its reference check spans every workspace, so the only scope that fits is Global, which blocks every other mutation in the fleet while the secret backend call runs. Your CAS fence (create and extend touch the credential, delete tombstones at the scanned version) already covers that race, so dropping the guard looks simplest, with a per-credential lock key as the alternative. Either way, keeping the secret deletion outside any guard matches what provider refresh does on main.
• One side effect: the old guard serialized creates for the same user. They now run in parallel and conflict on that credential touch, so a second concurrent create fails with "delegated credential changed during sandbox creation; retry". A bounded retry there is probably worth it.

I'm tracking this alongside the other #4147 follow-ups in #4307. Happy to help with the rebase if that's useful.

[grs]

@EmilienM Thanks for the heads-up! I have rebased and changed the lock scopes in line with your suggestions.