Skip to content

fix: resolve MTRL eval base-model ARN against the configured hub (not always SageMakerPublicHub)#6040

Open
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/mtrl-eval-private-hub-base-model-arn
Open

fix: resolve MTRL eval base-model ARN against the configured hub (not always SageMakerPublicHub)#6040
jam-jee wants to merge 1 commit into
aws:masterfrom
jam-jee:fix/mtrl-eval-private-hub-base-model-arn

Conversation

@jam-jee

@jam-jee jam-jee commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Issue

MTRL evaluation of a fine-tuned model via MultiTurnRLTrainer.attach(...) (or a ModelPackage) fails server-side inside the evaluation pipeline:

StepDetail(name='EvaluateFineTunedModel', status='Failed',
  failure_reason="ClientError: Failed to invoke sagemaker:CreateJob.
  Error Details: ResourceNotFound: Hub content with name mock-oss-test does not exist.")

Root cause

When the base model is resolved from a model package, its hub-content ARN is reconstructed from the package's BaseModel metadata. The backend populates only HubContentName + HubContentVersion (no HubContentArn), so the SDK rebuilds the ARN in _resolve_model_package_object — and that reconstruction was hardcoded to SageMakerPublicHub with an account-less (aws) owner:

# Base model always lives in SageMakerPublicHub (SAGEMAKER_HUB_NAME is for training recipes only)
base_model_arn = f"arn:aws:sagemaker:{region}:aws:hub-content/SageMakerPublicHub/Model/{name}/{version}"

For a model customized against a private/custom hub (SAGEMAKER_HUB_NAME), this yields a public-hub ARN that doesn't exist, and the eval pipeline's CreateJob fails with ResourceNotFound.

The bare-string / JumpStart-ID path does not have this bug — it already resolves via _resolve_jumpstart_model(..., get_sagemaker_hub_name()), which honors SAGEMAKER_HUB_NAME. Only the model-package / attached-job path ignored it.

This affects any customer who fine-tunes an MTRL model whose base model lives in a private/custom hub and then evaluates it via attach() or a ModelPackage — not just integration tests.

Fix

In _resolve_model_package_object, reconstruct the base-model ARN in the hub the model was actually customized against:

  • Use get_sagemaker_hub_name() (honors SAGEMAKER_HUB_NAME, still defaults to SageMakerPublicHub) instead of a hardcoded string.
  • Use the correct ARN owner: account-less aws for the public hub, the model package's own account for a private hub.

Verification

  • Reproduced against a live account: resolver now yields
    arn:aws:sagemaker:us-west-2:<acct>:hub-content/sdktest/Model/mock-oss-test/0.0.1,
    and describe-hub-content confirms that ARN is Available — so the pipeline's CreateJob finds the base model.
  • Unit tests: tests/unit/train/common_utils/test_model_resolution.py — 38 passed.
    • Added test_resolve_arn_construct_hub_content_arn_private_hub (private-hub case).
    • Pinned the existing test_resolve_arn_construct_hub_content_arn to the default public hub.

Notes

  • This unblocks the gpu_intensive nightly integ test test_evaluate_with_attached_trainer, which currently fails because mock-oss-test lives only in the private sdktest hub.
  • A complementary backend improvement (populate HubContentArn in the model package BaseModel block at registration time) would let the SDK skip reconstruction entirely — the SDK already prefers hub_content_arn when present. That is out of scope for this SDK change.

When evaluating a fine-tuned model via attach()/ModelPackage, the base
model's hub-content ARN is reconstructed from the model package's
BaseModel metadata (HubContentName + HubContentVersion) because the
backend does not populate HubContentArn. That reconstruction was
hardcoded to SageMakerPublicHub with an account-less ("aws") owner.

Models customized against a private/custom hub (SAGEMAKER_HUB_NAME) then
resolve to a public-hub ARN that does not exist, and the evaluation
pipeline's CreateJob fails server-side with:
  ResourceNotFound: Hub content with name <model> does not exist

Honor get_sagemaker_hub_name() when reconstructing the ARN, and use the
model package's own account for private hubs (public-hub content stays
account-less). The string/JumpStart-ID path already honored the hub via
_resolve_jumpstart_model; this aligns the model-package path with it.

Add a private-hub unit test and pin the existing test to the default hub.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant