Skip to content

enhance: support Azure SPN credentials via DefaultAzureCredential in IAM mode - #449

Open
sarveshls wants to merge 1 commit into
zilliztech:masterfrom
sarveshls:pr/azure-spn-default-credential
Open

sarveshls wants to merge 1 commit into
zilliztech:masterfrom
sarveshls:pr/azure-spn-default-credential

Conversation

@sarveshls

Copy link
Copy Markdown

When AZURE_CLIENT_SECRET is set, newAzureObjectStorageClient now selects
azidentity.NewDefaultAzureCredential before trying Workload Identity or
Managed Identity. This enables SPN auth using AZURE_CLIENT_ID +
AZURE_CLIENT_SECRET + AZURE_TENANT_ID without requiring a pod identity.

Priority mirrors DefaultAzureCredential's own chain order (EnvironmentCredential first).
Existing paths are unchanged.

A unit test verifies the client is built correctly when the three SPN vars are set.

/kind improvement

…IAM mode

Signed-off-by: Sarvesh Srivastava <sarvesh.srivastava@blackrock.com>
Signed-off-by: Sarvesh Srivastava <sarveshsrivastava72@gmail.com>
@github-actions

Copy link
Copy Markdown

CI commands

Comment any of these to rerun checks on this pull request:

Command Reruns
/rerun every failed workflow on the latest commit
/rerun-lint Lint
/rerun-mod-tidy Go Mod Tidy Check
/rerun-unit-test Unit Test
/rerun-e2e-local E2E Local
/rerun-e2e-objectstorage E2E Object Storage
/rerun-embed-minio-without-conditionwrite Embed MinIO Without ConditionWrite
/rerun-e2e-service E2E Service
/rerun-component-test Component Integration Test
/rerun-chaos Chaos Test
/rerun-service-stability-process Service Stability (Process)
/rerun-service-stability-docker Service Stability (Docker)
/rerun-operator Operator Test
/rerun-upgrade-local Upgrade Compat Local
/rerun-upgrade-objectstorage Upgrade Compat Object Storage
/rerun-upgrade-service Upgrade Compat Service

The ci-passed label is applied automatically once every required check passes.

@sarveshls

Copy link
Copy Markdown
Author

Related PRs applying the same credential selection pattern across the ecosystem:

# Repo PR URL Status
1 milvus-storage milvus-io/milvus-storage#742 Open
2 milvus-operator zilliztech/milvus-operator#558 Open
3 milvus milvus-io/milvus#54582 Open

@tinswzy

tinswzy commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

/assign @sarveshls

@codecov

codecov Bot commented Oct 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 84.10%. Comparing base (27e7032) to head (4872dd9).

Files with missing lines Patch % Lines
common/objectstorage/azure_object_storage.go 66.66% 0 Missing and 1 partial ⚠️

❌ Your patch check has failed because the patch coverage (66.66%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##           master     #449   +/-   ##
=======================================
  Coverage   84.09%   84.10%           
=======================================
  Files         225      225           
  Lines       31732    31734    +2     
=======================================
+ Hits        26686    26689    +3     
+ Misses       3775     3773    -2     
- Partials     1271     1272    +1     
Components Coverage Δ
Server 83.67% <ø> (ø)
Client 91.15% <ø> (+0.01%) ⬆️
Meta 85.58% <ø> (ø)
Common 88.33% <66.66%> (+<0.01%) ⬆️
Files with missing lines Coverage Δ
common/objectstorage/azure_object_storage.go 91.96% <66.66%> (+0.06%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tinswzy

tinswzy commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Must-fix issues introduced by this PR: 0
Merge recommendation: Mergeable as-is — this review found no must-fix issue introduced by this PR.

This PR adds a client-secret credential path for Azure IAM auth; the way that path is selected can silently switch to a different identity or break working Workload Identity setups, and the new test does not exercise the new branch.

Medium

common/objectstorage/azure_object_storage.go:212: Any AZURE_CLIENT_SECRET routes to DefaultAzureCredential, which silently falls back to another identity when tenant/client IDs are missing

  • Problem: The new branch only checks whether AZURE_CLIENT_SECRET is set, then calls azidentity.NewDefaultAzureCredential(nil) (DAC). If AZURE_TENANT_ID or AZURE_CLIENT_ID is empty, NewEnvironmentCredential fails. DAC does not return that error. It records it in an error reporter and keeps building the Workload Identity (WI), Managed Identity (MI), Azure CLI and azd sources, so credErr == nil at construction time. At runtime the reporter returns credentialUnavailable, and ChainedTokenCredential moves on to the next source (azidentity v1.6.0 default_azure_credential.go:74-82, environment_credential.go:89/93).
  • Failure: UseIAM=true with AZURE_CLIENT_SECRET set, but AZURE_TENANT_ID or AZURE_CLIENT_ID missing, on a pod or VM that has an MI attached (or an az CLI login) → startup succeeds, but Blob storage is accessed as the MI/CLI identity instead of the intended service principal (SPN). The result is either a 403 at runtime or reads and writes under the wrong identity, and the configuration error stays hidden.
  • Suggestion: Build the credential explicitly with azidentity.NewClientSecretCredential(tenantID, clientID, secret, nil). A missing value then fails at construction (fail-fast) instead of falling through the chain.

(raised by xiaocai2333, xaxys)

common/objectstorage/azure_object_storage.go:212: A working Workload Identity setup becomes a hard failure when AZURE_CLIENT_SECRET is also set

  • Problem: Before this change, the AZURE_FEDERATED_TOKEN_FILE check came first, so WI took priority. Now the AZURE_CLIENT_SECRET check runs first (the old if became else if at :214), and EnvironmentCredential is at the head of the DAC chain. An expired or mismatched secret produces an AuthenticationFailedError, not credentialUnavailable. ChainedTokenCredential stops on any error other than credentialUnavailable (chained_token_credential.go:99-102), so WI is never tried. The operator does not inject AZURE_* variables, so this combination can only come from the user's own environment, but such environments do exist.
  • Failure: A pod with valid WI (AZURE_FEDERATED_TOKEN_FILE) also has a stale AZURE_CLIENT_SECRET alongside AZURE_TENANT_ID/AZURE_CLIENT_ID → it works before the upgrade, and after it every object storage request fails authentication.
  • Suggestion: Using NewClientSecretCredential directly, as proposed in the item above, also fixes this. The secret path becomes explicit and deterministic, with no chain and no shadowing of other sources. Also decide on purpose and document which source wins when WI and secret variables are both present.

(raised by xiaocai2333, xaxys)

common/objectstorage/azure_object_storage_test.go:1063: The new test passes even with the new branch deleted

  • Problem: TestNewAzureClientDefaultAzureCredentialPath does not set AZURE_FEDERATED_TOKEN_FILE. If lines 212-213 of azure_object_storage.go are removed, execution falls into the MI branch (:220-230). There, NewManagedIdentityCredential makes no network call and returns no error, service.NewClient only builds a pipeline, CreateBucket=false skips the container check, and client.URL() returns a formatted string. All three assertions therefore hold on the code before this PR, so the test does not verify the new logic.
  • Suggestion: Assert on the credential that is actually built. For example, move credential selection into a small testable function and check that it returns *azidentity.ClientSecretCredential when the secret variables are set. Alternatively, set up a scenario whose outcome differs between branches.

(raised by xiaocai2333, xaxys)

Low

common/objectstorage/azure_object_storage_test.go:896: TestNewAzureObjectStorageClient_IAM_InvalidCredentials does not clear AZURE_CLIENT_SECRET, so the branch it tests now depends on the host environment

  • Problem: The test clears AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE but not AZURE_CLIENT_SECRET, which is now the only variable the new branch checks. If that variable is exported, the test takes the DAC path instead of the intended IAM path. EnvironmentCredential fails because tenant/client are empty, but DAC still tries WI, MI, Azure CLI and azd, and with CreateBucket enabled, GetProperties triggers a token request.
  • Failure: Running make unit-test on a machine with AZURE_CLIENT_SECRET exported and an active az CLI login → the test gets a real token with the local CLI identity and sends a request to testaccount.blob.core.windows.net. It no longer tests the intended path and adds external calls and latency, while assert.Error still passes, so the drift goes unnoticed.
  • Suggestion: Add t.Setenv("AZURE_CLIENT_SECRET", "") next to the other cleared variables.

(raised by xaxys)

@tinswzy

tinswzy commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

The SPN support is useful. I suggest the following changes before merging:

1. Preserve the existing Workload Identity priority and use an explicit SPN credential.

Currently, the PR checks AZURE_CLIENT_SECRET before AZURE_FEDERATED_TOKEN_FILE. A deployment with valid WI configuration and a leftover or expired secret would therefore switch from WI to SPN authentication and could fail after upgrading.

For backward compatibility, please use WI → SPN → MI. In the SPN branch, validate all three required variables and use ClientSecretCredential. This avoids DAC falling back to another identity when the SPN configuration is incomplete.

For example, the credential-selection block could be:

if os.Getenv("AZURE_FEDERATED_TOKEN_FILE") != "" {
    // Preserve the existing Workload Identity priority.
    cred, credErr = azidentity.NewWorkloadIdentityCredential(
        &azidentity.WorkloadIdentityCredentialOptions{
            ClientID:      os.Getenv("AZURE_CLIENT_ID"),
            TenantID:      os.Getenv("AZURE_TENANT_ID"),
            TokenFilePath: os.Getenv("AZURE_FEDERATED_TOKEN_FILE"),
        },
    )
} else if secret := os.Getenv("AZURE_CLIENT_SECRET"); secret != "" {
    tenantID := os.Getenv("AZURE_TENANT_ID")
    clientID := os.Getenv("AZURE_CLIENT_ID")
    if tenantID == "" || clientID == "" {
        return nil, errors.New(
            "SPN authentication requires AZURE_TENANT_ID, AZURE_CLIENT_ID and AZURE_CLIENT_SECRET",
        )
    }

    cred, credErr = azidentity.NewClientSecretCredential(
        tenantID, clientID, secret, nil,
    )
} else {
    opts := &azidentity.ManagedIdentityCredentialOptions{}
    if clientID := os.Getenv("AZURE_CLIENT_ID"); clientID != "" {
        opts.ID = azidentity.ClientID(clientID)
    }
    cred, credErr = azidentity.NewManagedIdentityCredential(opts)
}

if credErr != nil {
    return nil, credErr
}

This priority is a compatibility choice for Woodpecker; Azure's DAC uses Environment credentials first. See the Azure Identity credential chain guidance.

2. Add tests that verify credential selection and configuration errors.

Please cover WI, SPN, and MI selection, missing tenant/client IDs in the SPN branch, and WI taking priority when both WI and secret variables are present.

The new test only checks client construction and its URL with CreateBucket=false. These assertions would also pass through the old MI branch, so they do not verify SPN support. Extracting credential selection into a small helper would allow tests to assert the returned credential type.

Please also isolate the relevant Azure environment variables with t.Setenv so the tests do not depend on the host environment.

3. Document the supported authentication scenarios and selection order.

Please add configuration examples explaining when to use each option:

Authentication Typical scenario Configuration
Workload Identity AKS pods using ServiceAccount/OIDC federation AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_FEDERATED_TOKEN_FILE
SPN with client secret Environments without managed identity or workload federation AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_CLIENT_SECRET
Managed Identity Azure hosts with managed identity enabled Optional AZURE_CLIENT_ID for a user-assigned identity

Please state that these paths require useIAM=true, document the WI → SPN → MI precedence, and mention the required Blob data permissions, such as Storage Blob Data Contributor.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants