Support federated and temporary S3 / SigV4 credentials

[achtsnits]

Problem

Issue #43 asks how temporary credentials can be provided for protected assets, including:

• an OAuth2 access token for HTTP
• temporary credentials for S3

As far as I understand, the HTTP bearer-token use case can now be described by #44 using scheme chaining and RFC 8693 token exchange.

The published v1.2 example follows this flow:

OIDC identity token
        |
        v
OAuth2 token exchange
        |
        v
short-lived access token
        |
        v
protected asset

A shortened version of the example is:

{
  "auth:schemes": {
    "oidc": {
      "type": "openIdConnect",
      "openIdConnectUrl": "https://auth.example/.well-known/openid-configuration"
    },
    "token_exchange": {
      "type": "oauth2",
      "auth:refs": ["oidc"],
      "flows": {
        "tokenExchange": {
          "tokenUrl": "https://credentials.example/oauth/token",
          "subjectTokenType": "urn:ietf:params:oauth:token-type:id_token",
          "scopes": {
            "read:data": "Read data assets"
          }
        }
      }
    }
  },
  "assets": {
    "data": {
      "href": "https://storage.example/private-bucket/data.parquet",
      "auth:refs": ["token_exchange"]
    }
  }
}

The remaining question from #43 therefore seems to be the S3 case: how a federated identity maps to temporary S3 / SigV4 credentials using:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_SESSION_TOKEN

There seem to be at least two useful S3 flows worth mentioning:

1. Temporary S3 credentials from an STS

The identity token is exchanged for a new set of temporary S3 credentials:

OIDC JWT
   |
   v
STS / AssumeRoleWithWebIdentity
   |
   v
AccessKeyId + SecretAccessKey + SessionToken
   |
   v
S3 / SigV4
   |
   v
protected asset

Conceptually:

{
  "auth:schemes": {
    "identity": {
      "type": "openIdConnect",
      "openIdConnectUrl": "https://identity.example/.well-known/openid-configuration"
    },
    "storage": {
      "type": "s3",
      "auth:refs": ["identity"],
      "scheme": "sigv4",
      "credentialExchangeUrl": "https://storage.example/sts"
    }
  }
}

The field names are only examples. credentialExchangeUrl would point to a service that accepts the referenced identity and returns temporary S3 credentials.

From past discussions with@alukach, my understanding is that this is the flow Source Cooperative is following. It provides an AWS-compatible AssumeRoleWithWebIdentity flow, so standard AWS credential providers can be used with AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, and a custom STS endpoint.

2. Direct federated SigV4

An S3-compatible service can also trust the federated token directly, without another STS exchange.

For example, the referenced identity token can be carried in the standard AWS session-token field, while still using the standard three-part temporary credential interface and standard SigV4 requests:

AWS_ACCESS_KEY_ID=<subject or other stable identifier>
AWS_SECRET_ACCESS_KEY=<deterministically derived from OIDC JWT>
AWS_SESSION_TOKEN=<OIDC JWT>

The additional convention is how the AccessKeyId and SecretAccessKey are constructed from the referenced identity token. The derived SecretAccessKey is then used by the client following the standard SigV4 signing process.

The flow becomes:

OIDC / workload identity
  |
  v
S3 SigV4
  |
  v
protected asset

Conceptually:

{
  "auth:schemes": {
    "identity": {
      "type": "openIdConnect",
      "openIdConnectUrl": "https://identity.example/.well-known/openid-configuration"
    },
    "storage": {
      "type": "s3",
      "auth:refs": ["identity"],
      "scheme": "sigv4"
    }
  }
}

Here, there is no credential exchange endpoint. The referenced short-lived JWT is used directly to construct the SigV4 credential set: the client puts the raw JWT into AWS_SESSION_TOKEN and deterministically derives AWS_SECRET_ACCESS_KEY according to the agreed profile before passing the credentials to a standard S3 signer. Because anyone possessing the JWT can derive the same signing key, this provides request-level SigV4 integrity but no independent proof of possession beyond the bearer JWT itself.

This resembles Keycloak federated client authentication in that a workload uses a short-lived, externally issued JWT as its client assertion instead of managing a static client secret. Keycloak, however, validates the assertion at its token endpoint and issues a separate access token. Here, the storage authorization service validates the original JWT on every request and additionally verifies the SigV4 signature derived from it. This is the model currently used by us and planned for the EarthCODE data orchestration layer. It avoids a separate STS exchange and allows the storage authorization service to make a single local authorization decision based on the validated identity and current resource policies.

Proposal

Extend the s3 authentication scheme so it can describe:

1. S3 / SigV4 authentication using temporary credentials.
2. AccessKeyId, SecretAccessKey and SessionToken.
3. How these credentials relate to another authentication scheme through auth:refs.
4. Whether the referenced identity must first be exchanged at an S3 credential endpoint.
5. Whether the referenced identity token can instead be used directly as the SigV4 session token.

The exact fields above are only examples.

The goal is to let a generic STAC client discover whether it needs to:

identity -> STS -> temporary S3 credentials -> SigV4

or can use:

identity -> SigV4 directly

The S3 service can still decide how the identity is validated, how the signing secret is derived or provided, and how authorization is performed.

Does this distinction make sense, and would you agree that the s3 authentication scheme should cover both flows? If so, we could prepare a PR based on this.