fix: resolve MTRL eval base-model ARN against the configured hub (not always SageMakerPublicHub)#6040
Open
jam-jee wants to merge 1 commit into
Open
Conversation
When evaluating a fine-tuned model via attach()/ModelPackage, the base
model's hub-content ARN is reconstructed from the model package's
BaseModel metadata (HubContentName + HubContentVersion) because the
backend does not populate HubContentArn. That reconstruction was
hardcoded to SageMakerPublicHub with an account-less ("aws") owner.
Models customized against a private/custom hub (SAGEMAKER_HUB_NAME) then
resolve to a public-hub ARN that does not exist, and the evaluation
pipeline's CreateJob fails server-side with:
ResourceNotFound: Hub content with name <model> does not exist
Honor get_sagemaker_hub_name() when reconstructing the ARN, and use the
model package's own account for private hubs (public-hub content stays
account-less). The string/JumpStart-ID path already honored the hub via
_resolve_jumpstart_model; this aligns the model-package path with it.
Add a private-hub unit test and pin the existing test to the default hub.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
MTRL evaluation of a fine-tuned model via
MultiTurnRLTrainer.attach(...)(or aModelPackage) fails server-side inside the evaluation pipeline:Root cause
When the base model is resolved from a model package, its hub-content ARN is reconstructed from the package's
BaseModelmetadata. The backend populates onlyHubContentName+HubContentVersion(noHubContentArn), so the SDK rebuilds the ARN in_resolve_model_package_object— and that reconstruction was hardcoded toSageMakerPublicHubwith an account-less (aws) owner:For a model customized against a private/custom hub (
SAGEMAKER_HUB_NAME), this yields a public-hub ARN that doesn't exist, and the eval pipeline'sCreateJobfails withResourceNotFound.The bare-string / JumpStart-ID path does not have this bug — it already resolves via
_resolve_jumpstart_model(..., get_sagemaker_hub_name()), which honorsSAGEMAKER_HUB_NAME. Only the model-package / attached-job path ignored it.This affects any customer who fine-tunes an MTRL model whose base model lives in a private/custom hub and then evaluates it via
attach()or aModelPackage— not just integration tests.Fix
In
_resolve_model_package_object, reconstruct the base-model ARN in the hub the model was actually customized against:get_sagemaker_hub_name()(honorsSAGEMAKER_HUB_NAME, still defaults toSageMakerPublicHub) instead of a hardcoded string.awsfor the public hub, the model package's own account for a private hub.Verification
arn:aws:sagemaker:us-west-2:<acct>:hub-content/sdktest/Model/mock-oss-test/0.0.1,and
describe-hub-contentconfirms that ARN isAvailable— so the pipeline'sCreateJobfinds the base model.tests/unit/train/common_utils/test_model_resolution.py— 38 passed.test_resolve_arn_construct_hub_content_arn_private_hub(private-hub case).test_resolve_arn_construct_hub_content_arnto the default public hub.Notes
gpu_intensivenightly integ testtest_evaluate_with_attached_trainer, which currently fails becausemock-oss-testlives only in the privatesdktesthub.HubContentArnin the model packageBaseModelblock at registration time) would let the SDK skip reconstruction entirely — the SDK already prefershub_content_arnwhen present. That is out of scope for this SDK change.