Resolve merge conflict - #6161
Open
zhaoqizqwang wants to merge 50 commits into
Open
Conversation
Adds an automated PR review using anthropics/claude-code-action@v1 running on Amazon Bedrock. Inference stays in-account and is CloudTrail-audited; no external API key is required. Reuses the existing collaborator gate from pr-checks-master.yml so that collaborator PRs auto-run and fork/external PRs require manual approval before any secret or the Bedrock role is exposed (safe under pull_request_target).
…#6040) When evaluating a fine-tuned model via attach()/ModelPackage, the base model's hub-content ARN is reconstructed from the model package's BaseModel metadata (HubContentName + HubContentVersion) because the backend does not populate HubContentArn. That reconstruction was hardcoded to SageMakerPublicHub with an account-less ("aws") owner. Models customized against a private/custom hub (SAGEMAKER_HUB_NAME) then resolve to a public-hub ARN that does not exist, and the evaluation pipeline's CreateJob fails server-side with: ResourceNotFound: Hub content with name <model> does not exist Honor get_sagemaker_hub_name() when reconstructing the ARN, and use the model package's own account for private hubs (public-hub content stays account-less). The string/JumpStart-ID path already honored the hub via _resolve_jumpstart_model; this aligns the model-package path with it. Add a private-hub unit test and pin the existing test to the default hub.
…6041) Make ModelBuilder.list_deployment_configs / get_deployment_config / set_deployment_config work for fine-tuned (model-customization) models in addition to base/JumpStart models, dispatching internally on _is_model_customization(). For fine-tuned models the recipe's published HostingConfigs are the source of truth; selection is by instance type (recipe configs are largely unnamed). The base "deployment config" vs recipe "hosting config" distinction is internal and does not surface: fine-tuned configs are normalized to a shape compatible with the base response (DeploymentConfigName plus a nested DeploymentArgs block, plus BenchmarkMetrics/AccelerationConfigs and an additive IsDefault flag). The normalized fine-tuned shape always populates the DeploymentArgs keys (None when unset); the base response may omit unset keys (its serializer drops empty slots), so the fine-tuned shape is a superset — callers consuming both pathways should use .get() for the optional keys (documented on the methods). set_deployment_config applies the whole matching config (image, environment, compute requirements) at build time. A caller-provided image_uri still takes precedence over the config's ImageUri (documented). Both branches fail fast on bad input: the fine-tuned branch requires instance_type and raises on an unpublished or ambiguous instance; the base branch requires config_name AND instance_type and now also rejects an unpublished config name or an instance the config does not support (validated against JumpStart metadata) instead of silently recording a no-op selection. Instance-type matching honors the config's full offered set, not only its default. A base config is a multi-instance bundle: list_deployment_ configs(instance_type=X) filters against each config's supported-instance metadata (supported_inference_instance_types) and materializes matched configs FOR X, so a config that supports X but defaults to another instance is neither discarded nor materialized at the wrong instance. Recipe hosting configs are per-instance bundles by contract, but SupportedInstanceTypes, when present, is honored for selection and filtering. list/get/set agree for the same selection: get_deployment_ config() materializes the pinned instance (matching what list returns and what build deploys), including when the pinned instance came from SupportedInstanceTypes (differs from the config's default). The pinned instance is preserved through build. Configs published with only DefaultInstanceType are matched consistently end to end. The build env merge tolerates a config that publishes an explicit null Environment. Config discovery (recipe-level with a top-level HostingConfigs fallback) is centralized in _extract_hosting_configs_from_hub() and used by BOTH the selection API and the build path, so a top-level config that is listable/selectable is also applied at build. Nova models are routed to their dedicated build path first (per-tier SMI validation + Nova env precedence), so the top-level fallback never diverts them. An explicit selection stores a deep copy of the raw config, is applied exactly at build (or raises if no longer published), and returned configs are copied so caller mutation cannot corrupt internal state. This is the SDK half of a paired change; the SageMaker agent model-deployment skill consumes this unified API. Internal CR: https://code.amazon.com/reviews/CR-289903643
Replace ci-health-v3.yml and the standalone gpu-integ-tests.yml with two workflows, and move the GPU integ tests into the V3 suite. - ci-health-v3-master.yml: unit-test-v3, canaries-v3-master, gpu-integ-tests-master (us-west-2 + us-east-1), and import-model-integ-tests. A report-result job ANDs the GPU + import-model jobs into a single per-run GpuIntegMasterRunFailure metric emitted to us-west-2. - ci-health-v3-release.yml: canaries-v3-release, gpu-integ-tests-release (us-west-2 + us-east-1), with report-result ANDing the two region jobs into GpuIntegReleaseRunFailure. Both run once a day (no gate / skip-on-success). Each region job pair triggers the same CodeBuild project name (region differs only by credentials and the region-override buildspec), and the AND-ed run-level metric preserves us-east-1 GPU coverage that a us-west-2-only per-project alarm cannot see. Only scheduled runs emit the metrics; manual dispatch runs are excluded.
* Fix role issue in mtrl integ tests * Fix role issue in mtrl integ tests --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…#6076) The OSS-import path in bedrock-modelbuilder-deployment.ipynb hardcoded job_name="import-llama-finetuned" and imported_model_name="llama-3-8b-finetuned". Bedrock's CreateModelImportJob requires a unique job name, and completed import jobs cannot be deleted, so running the notebook a second time in the same account/region fails with "The provided job name is currently in use." Append an int(time.time()) suffix to both names so the example is repeatable. time is already imported in the notebook. Co-authored-by: hrehard <hrehard@amazon.com>
* cleanup: use a single logger in base_trainer * fix: update recipe resolver unit tests * Release 3.16.0 (2026-07-15): Bump VERSION files and internal dependency pins to 3.16.0 / 2.16.0 / 1.16.0. Update CHANGELOGs across root and submodules (core, train, serve, mlops). * fix: use absolute path in data mixing recipe path construction --------- Co-authored-by: Syed Jafri <syedjfr@amazon.com>
mtrl_finetuning_example_notebook_v3_prod.ipynb calls pprint() to display the trainer's default hyperparameters but never imports it, so that cell raises NameError: name 'pprint' is not defined. Add 'from pprint import pprint'. Co-authored-by: hrehard <hrehard@amazon.com>
…delbuilder-deployment example (aws#6091) * fix: resolve Nova model artifacts via TrainingJob.get() in bedrock deploy example In bedrock-modelbuilder-deployment.ipynb, after sft_trainer.train(wait=True) the example printed sft_trainer._latest_training_job.model_artifacts.s3_model_artifacts. For Nova training jobs the DescribeTrainingJob API returns no ModelArtifacts, and the trainer's cached _latest_training_job is not re-fetched -- so model_artifacts stays Unassigned and the cell raises 'AttributeError: Unassigned object has no attribute s3_model_artifacts', halting the notebook before deploy. Re-fetch the completed job with TrainingJob.get(), which synthesizes the artifacts path from output_data_config (same pattern the earlier get() cell already uses). * docs: shorten comment in bedrock deploy example (public notebook) --------- Co-authored-by: hrehard <hrehard@amazon.com>
The PR-check `unit-tests` job invoked per-submodule CodeBuild projects named `sagemaker-python-sdk-ci-<submodule>-unit-tests`. Those projects were created manually (no CloudFormation tags) and are not managed by the SageMakerMLFPySDKInfraCDK pipeline, so they drifted stale: they still run `tox ... --cov=.` (last modified 2026-06-15) even though the merged and deployed buildspec CR changed this to `--cov=sagemaker`. As a result PR coverage kept reporting the old test-file-inflated numbers and never reflected the fix. Point the job at the single CDK/pipeline-managed project `sagemaker-python-sdk-ci-health-unit-test-v3` (createCIUnitV3BuildSpec), driven by the `SUBMODULE` env var, exactly as the ci-health workflow invokes it. This project carries the deployed `--cov=sagemaker` buildspec, so PR coverage now tracks the intended product-only measurement, and the CI wiring stays in sync with the CDK going forward. The manual per-submodule `-unit-tests` projects can be retired separately.
The review job runs under pull_request_target (trusted context with the
Bedrock role + secrets). It was checking out the fork's head SHA, which
actions/checkout now refuses ('pwn request' guard) — so the workflow failed on
every fork PR (~90% of PRs), e.g. run 29965762436 on aws#6083.
Rework to the safe diff-only pattern (mirrors aws/*-agentcore pr-security-review):
- Check out the BASE repo, never fork head code — untrusted PR code is never
executed in the privileged job.
- Fetch the PR diff via the API into /tmp/pr.diff as read-only ground truth.
- Remove Bash from allowedTools; Claude reads the diff + uses Read/Grep/Glob
against the trusted base tree for context. Skip cleanly on empty diffs.
Does NOT use allow-unsafe-pr-checkout, which would expose secrets to fork code.
Sets include_fix_links: false on claude-code-action. The 'Fix this' deep-links open Claude Code with issue context — useless for external contributors (~90% of PRs here) and just noise in review comments.
…tes (aws#6081) Integ tests run under pytest -n auto (dozens of xdist workers). Many resolve/ validate an IAM execution role via resolve_and_validate_role, which calls the low-TPS iam:SimulatePrincipalPolicy API. Under concurrent load IAM throttles it (ClientError: Throttling / Rate exceeded), failing tests during setup or build. Add an identical test-harness mitigation to the serve, train, and mlops integ conftests: - autouse session fixture sets adaptive retries via AWS_RETRY_MODE / AWS_MAX_ATTEMPTS env vars, so EVERY boto client in the worker inherits them (including IAM clients the resolver builds from an explicitly-passed Session, which the previous serve-only DEFAULT_SESSION approach missed). - pytest_runtest_makereport converts residual SimulatePrincipalPolicy throttling into a skip (setup + call phases). Throttling on any other op still fails loud. train gets a new parent tests/integ/conftest.py covering train/, ai_registry/, and jumpstart/. sagemaker-core integ is unaffected (no resolver call path).
…ws#6094) * test(integ): let exhausted IAM throttling fail instead of skipping The prior mitigation converted a SimulatePrincipalPolicy throttle that survived the adaptive retries into a skipped test (pytest_runtest_makereport). That hid a persistent rate-limit regression: a genuinely throttled run would silently drop out of the results instead of showing up as a failure. Keep the autouse adaptive-retry fixture (it still absorbs transient bursts), but remove the skip conversion and its now-unused helper/constants. Throttling that exhausts the retry budget now fails the test loudly so the regression is visible. * test(integ): remove throttle skip hook from train & mlops; fix serve merge artifacts The master merge into this branch reintroduced the pytest_runtest_makereport skip hook in the train and mlops integ conftests (both landed by aws#6081), so a SimulatePrincipalPolicy throttle surviving the retries would still be silently skipped there. Remove the hook and its now-unused helper/constants from both, so exhausted throttling fails loudly in every suite. Also fix two artifacts the merge left in the serve conftest: - restore the fixture teardown that resets AWS_RETRY_MODE / AWS_MAX_ATTEMPTS (the 'previous' dict was captured but never restored -> unused-variable lint and env leak across the session); - update the stale docstring that still described the removed _configure_default_boto_retries / DEFAULT_SESSION approach.
…sts (aws#6095) The feature processor to_pipeline integ tests hardcoded fixed pipeline names (pipeline-name-01, pipeline-name-lf-01) and thus shared fixed S3 paths (s3://.../<pipeline_name>/function/payload.pkl). Combined with the asymmetric-signing scheme introduced in PR aws#5816 (each to_pipeline call generates a fresh ECDSA key pair, overwrites the signed payload, and pins the matching public key into the pipeline env), concurrent CI builds on the same account overwrite each other's payloads. A running execution then verifies a payload signed by a different build's key, producing DeserializationError: "Integrity check for the serialized function or data failed" and pipeline execution status Failed. Generate unique pipeline names via unique_name_from_base so each test run and build uses an isolated S3 prefix, and restore cleanup_pipeline in the finally blocks to avoid resource leakage.
…aws#6092) * fix(train): Fall back to public hub when private hub lacks base model When resolving a model package's base model, the reconstructed hub-content ARN honored SAGEMAKER_HUB_NAME but never verified the base model actually existed in that private hub, and never fell back. If the private hub had not mirrored the base model (or its ModelReference was cleaned up), server-side CreateTrainingJob failed with "Hub content ... does not exist", breaking evaluation integ tests (benchmark, custom scorer, and LLM-as-judge, including custom scorer with a built-in metric). Verify the base model exists in the configured private hub and fall back to SageMakerPublicHub when it is absent, mirroring the existing fallback in _resolve_jumpstart_model. Base models are always published to the public hub, so this keeps evaluation working regardless of private-hub contents. The public-hub path is unchanged and does no extra lookups. * nit: update docstring
* fix(iam): scope repo-level ECR actions to prevent false deny in preflight validation
Split ecr_policy statements in training, serving, and hyperpod role types
so that only ecr:GetAuthorizationToken (account-level) remains under
Resource: "*". The repository-level actions (BatchGetImage,
GetDownloadUrlForLayer, BatchCheckLayerAvailability) are now scoped to
arn:aws:ecr:*:*:repository/*, which excludes them from
_get_smoke_test_actions. This prevents SimulatePrincipalPolicy from
returning implicitDeny for roles that correctly scope ECR permissions to
specific repo ARNs (least privilege), fixing the regression that blocked
deploys/training/pipelines for those customers.
* fix: support s3 uri in FrameworkProcessor
* add local dependencies to source_dir
* fix: make dependencies optional in FrameworkProcessor code-packaging helpers
_package_code and _pack_and_upload_code declared dependencies as a
required positional arg, but the unit tests (and any direct callers)
invoke them without it, causing:
TypeError: _package_code() missing 1 required positional argument: 'dependencies'
Move dependencies to the end of both signatures with a None default so
it is optional, matching the public run() signature. Also apply black
formatting and remove unused imports/vars in test_processing.py.
* fix: mount install_requirements.py for S3 source_dir processing jobs
When source_dir is an S3 URI, /opt/ml/processing/input/code maps to the
user's (possibly read-only) S3 location, which does not contain the
managed install_requirements.py helper. The helper was uploaded to a
separate managed prefix that was never mounted into the container, yet
the generated runproc script hardcoded
python3 /opt/ml/processing/input/code/install_requirements.py
so any S3 source_dir bundle containing a requirements.txt failed at
runtime with a missing-file error.
Mount the managed helper via a dedicated 'aux' ProcessingInput at
/opt/ml/processing/input/aux (never writing into the user's bucket) and
parameterize the generated scripts to reference install_requirements.py
from the correct directory: /input/code for a local source_dir
(unchanged) and /input/aux for an S3 source_dir.
Update test_pack_and_upload_code_with_s3_source_dir_creates_code_input to
assert the aux mount is created (it previously locked in the broken
layout) and add regression tests verifying the generated scripts point at
the mounted helper path.
Also use a context manager for the helper file read.
---------
Co-authored-by: Mohamed Zeidan <81834882+mohamedzeidan2021@users.noreply.github.com>
Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
Co-authored-by: Mohamed Zeidan <81834882+mohamedzeidan2021@users.noreply.github.com>
…s#6026) - Add use_batch_write_record=False flag to ingest_dataframe() (Proposal C) - Implement _ingest_batch_write() with 25 records per API call - Map partial failures (response.errors) back to specific row indices - Add list_records() utility function with pagination support - Export list_records from feature_store __init__.py Tested: 50 integration tests + 17 unit tests
…6014) * feat(feature-store): Add lineage registration to DatasetBuilder Add register_as_dataset parameter to DatasetBuilder.create() that registers the built dataset as a HubContent with source feature group ARNs in ContentMetadata. This enables auto-lineage between Feature Groups and Training Jobs via SM Dataset (HubContent) as bridge. Changes: - Add register_as_dataset flag to DatasetBuilder.create() - Collect source FG ARNs from base and merged feature groups - Call HubContent.import_hub_content() with sourceFeatureGroups, extractionMethod, and athenaQueryExecutionId metadata - Graceful fallback on AccessDenied or generic errors (logs warning) - Add 10 unit tests covering opt-in, ARN collection, dedup, DataFrame path, HubContent invocation, error handling * test: skip flaky Lake Formation integ test The test_to_pipeline_and_execute_with_lake_formation test fails due to test role permissions being modified from the console. Skipping until the LF environment is reconfigured. Not related to this PR's changes. Agreed with @bhhalim and @SREEDEVI to skip for now. --------- Co-authored-by: Vishakha Nerkar <vnerkar@amazon.com> Co-authored-by: rsareddy0329 <rsareddy0329@gmail.com>
…erage (aws#6056) `after_n_builds: 4` made Codecov wait for four coverage uploads before finalizing a report. But this repo's CI runs unit tests only for the submodules a PR changes (detect-changes), so a PR touching 1-3 submodules produces fewer than four uploads. Codecov then waits forever and never posts a report or status. Observed on PR aws#6055 (changed only sagemaker-mlops): the mlops unit-test job uploaded coverage successfully, but Codecov reported state=error, sessions=0 and posted no codecov/project or codecov/patch status, because it was still waiting for three more uploads that never come. Drop the fixed count and rely on `wait_for_ci: true`, which finalizes the report once CI completes using whatever uploaded — correct for this repo's dynamic per-submodule CI.
…sform() (aws#6110) Transformer.transform() submitted the CreateTransformJob request successfully, then rebuilt a local TransformJob resource from the same request dict. That dict contains a "tags" key (Tags is a member of the CreateTransformJobRequest shape), but the TransformJob resource model has no tags field and sets extra="forbid", so the constructor raised a pydantic ValidationError after the job was already created on SageMaker, leaving the caller without a job handle. Pop "tags" from the transformed dict before constructing TransformJob, mirroring the existing ProcessingJob fix (aws#5459). Add a regression test.
…#6112) * fix(serve): repack source_code for image_uri/ModelTrainer builds ModelBuilder classified any image_uri build with a model artifact and source_code (no model/inference_spec) as passthrough and dropped the inference code, so build() produced a model.tar.gz without code/ and register() yielded a package that could not serve. This regressed the classic v2 Model(model_data, entry_point, source_dir) repack behavior. In _build_for_passthrough(), distinguish a pure image-only deployment from an "image + model artifact + source_code" build. For the latter, keep the source code and bridge model_path to s3_model_data_url so the existing _upload_code(repack=True) path repacks the code into the model artifact, producing a self-contained tarball. Pure-image (Nova/BYOC) passthrough behavior is unchanged. ModelTrainer builds are covered too since they are normalized to model_path before validation. Also pass script_dependencies (a list derived from SourceCode) to repack_model instead of the deprecated self.dependencies auto-detect dict, which repack_model would otherwise iterate as filesystem paths. * fix(serve): honor source_code for image_uri/model-server builds ModelBuilder classified any image_uri build without an in-memory model or inference_spec as passthrough, ignoring source_code. _build_for_ passthrough() then cleared source_dir/entry_point, so custom inference code was silently dropped: no repack occurred, the container pointed at the raw model artifact, register()/deploy() produced a model with no serving code. Regressed from v2 Model(model_data, entry_point, source_dir) + _RepackModelStep; introduced by aws#5969 (commit ebeb54c) which added the passthrough nulling. Fix the passthrough classification at its source: do not treat a build as passthrough when source_code is set (both 1P and non-1P image branches). Such builds route to the normal model-server path and repack the code into the artifact. Also: - Relax the _build_for_model_server guard to accept s3_model_data_url (a prebuilt training artifact) alongside model/MLflow/inference_spec. - In _upload_code repack, pass script_dependencies (a list) instead of the deprecated self.dependencies auto-detect dict, which repack_model would otherwise iterate as filesystem paths. - Update _build_for_model_server unit tests to set s3_model_data_url so the missing-parameter cases still validate. * nit: clean up long comments not necessary for customer * fix(serve): repack source_code inside passthrough path Restore v2-parity repack behavior for image_uri + model artifact + source_code builds by repacking inside _build_for_passthrough(), instead of gating these builds out of passthrough into the model-server path. Routing such builds to _build_for_model_server exposed pre-existing gaps in the server builders (e.g. _build_for_djl reads hf_model_config, which is unset when model is None) and diverged from the path that worked before serve 1.15.0. Repacking within passthrough matches the 1.12.0 behavior and keeps the fix self-contained. - _build_validations: drop the source_code passthrough exclusion so these builds stay on the passthrough path - _build_for_passthrough: when source_code and a model artifact are both present, bridge model_path to s3_model_data_url and repack into the artifact; otherwise keep the pure image-only passthrough behavior - _build_for_model_server: revert the s3_model_data_url guard relaxation - tests: revert the now-unneeded servers guard test changes Verified end-to-end (build -> deploy -> invoke) for both the ticket scenario (image_uri + model_server=DJL_SERVING + source_code + s3_model_data_url) and the aws#6105 scenario (no model_server). * test(serve): cover source_code repack in passthrough build Add unit tests for _build_for_passthrough repacking source_code into the model artifact (and bridging model_path to s3_model_data_url), a regression test for pure image-only passthrough, and a test that _upload_code repack passes script_dependencies instead of the deprecated dependencies dict. Add a build-only integ test that verifies build() repacks code/inference.py into the artifact and wires up the script-mode env vars. * fix(serve): warn on unrepackable source_code; address review nits - Warn (instead of silently dropping) when source_code has an entry_script but no source_dir, so the code cannot be repacked into the artifact - Drop the redundant str() guard on model_path for consistency - Add unit tests: is_repack() is True after the repack branch, and the entry_script-without-source_dir warning path
aws#6116) * fix(train): assign SDK-managed channels to instance groups on heterogeneous clusters ModelTrainer injects code/sm_drivers/recipe channels that users cannot configure. On a heterogeneous cluster, SageMaker enforces an all-or-nothing rule: if any channel is assigned to instance groups, every channel must be. So setting instance_group_names on any user channel caused CreateTrainingJob to fail with "Some channels have assigned instance groups ... while others not: [sm_drivers, code]", making instance_group_names unusable with ModelTrainer. When a user assigns instance groups to any channel, assign the SDK-managed channels to the full set of instance groups (code/drivers must be present on every node). Behavior is unchanged when instance groups are not in use. Fixes aws#6089 * chore(train): shorten inline comments --------- Co-authored-by: Mohamed Zeidan <zeidmo@amazon.com>
…lient (aws#6107) SageMakerClient.__init__ built the "sagemaker" control-plane client from a fresh default botocore session (botocore.session.get_session()) instead of the caller-provided session. The custom data loader for the pre-GA Job APIs needs a botocore session to attach to, and the new session it created silently dropped the caller's credentials/profile. Since SageMakerClient is a singleton, that default-credential client was reused process-wide, so control-plane calls went out under the ambient default AWS profile. When the execution role lived in a different account than the default profile, CreateTrainingJob failed with "RoleArn: Cross-account pass role is not allowed". This regressed V2 behavior and was introduced in the MTRL launch (aws#5919, first shipped in v3.13.0). Register the custom data loader on the caller session's underlying botocore session (session._session) and build the sagemaker client from that session, so it keeps the caller's credentials while still loading the custom service model. The other clients already used the passed session and are unchanged.
…s#6115) * Fix gpu integ test failure due to outdated MPG * Fix: fix agent_run_time inference from the attached input trainer --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…ws#5328) Bumps [torch](https://github.com/pytorch/pytorch) from 2.0.1+cpu to 2.8.0. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](https://github.com/pytorch/pytorch/commits/v2.8.0) --- updated-dependencies: - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: lucasjia-aws <lucasjia@amazon.com>
Bumps [torch](https://github.com/pytorch/pytorch) from 2.7.0 to 2.8.0. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](pytorch/pytorch@v2.7.0...v2.8.0) --- updated-dependencies: - dependency-name: torch dependency-version: 2.8.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: lucasjia-aws <lucasjia@amazon.com>
…ws#6119) Bumps [black](https://github.com/psf/black) from 24.3.0 to 26.3.1. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@24.3.0...26.3.1) --- updated-dependencies: - dependency-name: black dependency-version: 26.3.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: lucasjia-aws <lucasjia@amazon.com>
…runs (aws#6098) A concurrency search / magic-list sweep (--search-recipe or --concurrency 1,10,100) does not produce a single top-level profile_export_aiperf.json. The benchmark container instead ships search_history.json at the artifact root plus per-trial profile exports nested in subdirs. The existing reader matched profile_export_aiperf.json by filename suffix, so for a sweep it would either fail (no such root file) or silently return one arbitrary level's per-trial metrics as if they were the whole benchmark. BenchmarkResult.from_s3 now checks for search_history.json first and, when present, parses the sweep outcome into a new BenchmarkSearchResult (winning swept value from boundary_summary.feasible_max, the first constraint breach, and the raw history). The single-run path is unchanged. Adds BenchmarkSearchResult (exported), a BenchmarkResult.search field + is_search property, and unit tests covering the sweep layout, the per-trial-name collision, null boundary summaries, and the single-run regression guard.
* fix(sagemaker-core): remove dev-only endpoint override and fix client singleton pinning
Remove the temporary SAGEMAKER_ENDPOINT / SAGEMAKER_RUNTIME_ENDPOINT /
SAGEMAKER_STAGE overrides and the custom botocore service-model loader from
SageMakerClient. These were internal beta/gamma testing shims (marked
"# TODO: Remove post-launch") added while the generic Job APIs were not yet in
public botocore. Those APIs (CreateJob/DescribeJob/ListJobs) now ship in public
botocore, so the standard session.client("sagemaker", ...) path is used and the
env-driven endpoint redirection is no longer read by the SDK.
Also fix SingletonMeta so SageMakerClient is keyed by (session, region) instead
of by class alone. Previously the first instance was pinned process-wide, so a
later call with a different session/region silently reused the original client.
Parameterless SageMakerClient() calls still share a single instance.
Harden the container-driver env logger to mask values that look like AWS
credential material (access key IDs, session tokens) even when the key name
contains none of SENSITIVE_KEYWORDS.
* fix(sagemaker-train): mask credential-shaped values in container env logging
The startup env logger masked only by key name, so a sensitive value under a
key name without a SENSITIVE_KEYWORDS substring was logged verbatim. Add
value-based detection for AWS credential material (access key IDs, session
tokens) so such values are masked regardless of key name.
* fix(sagemaker-core): address review feedback on client singleton
Make SageMakerClient._singleton_key tolerant of extra constructor
arguments by accepting *args/**kwargs, so callers passing additional
keywords (such as the legacy service_name=) no longer raise a
TypeError during key computation on cache-hit paths.
Document why id(session) is a stable cache key: the cached instance
retains the session reference, preventing garbage collection and id
reuse while the entry lives.
Add unit tests for the singleton keying and reset(): distinct instance
per (session, region), reuse across repeated no-arg calls, reuse for an
identical session/region, and reset() clearing all keyed entries for
the class.
Rework the env-logging test assertion to compare the logged arguments
exactly instead of a URL substring check, removing the CodeQL
incomplete-URL-sanitization finding on the test file.
* revert(sagemaker-core): keep SageMakerClient singleton keyed by class
Revert the (session, region) singleton keying introduced earlier in this
PR. The security concern behind Finding 3 (a poisoned endpoint being
pinned) is already resolved by removing the SAGEMAKER_ENDPOINT override
in Finding 1, so there is no attacker-controlled endpoint left to pin.
Per team design, the SDK operates with a single session/region per
process, so a class-keyed singleton is the intended behavior.
Restores the original SingletonMeta, SageMakerClient.reset(), and drops
the singleton keying unit tests. Finding 1 (endpoint override removal)
and Finding 5 (env-log value masking) are unchanged.
* fix(sagemaker-core): bump boto3 floor to >=1.43.20 for Job APIs
Removing the custom sample-model loader makes the control-plane client a
stock session.client("sagemaker"), so the generic Job APIs must be present
in the resolved botocore. CreateJob/DescribeJob/ListJobs first ship in
botocore 1.43.20; the previous floor (boto3>=1.42.2) allowed botocore
versions without them, which would regress Job.create()/get()/get_all()
at runtime.
Raise the pin to boto3>=1.43.20 (which requires botocore>=1.43.20) to
guarantee the Job APIs are available.
* revert(sagemaker-train): drop container env-log value masking
Remove the value-based masking added earlier in this PR for the container
-driver env logger. The behavior it targeted was raised as a forensic
signal rather than a vulnerability, and comprehensive credential masking
in logs is handled by the platform-side CloudWatch Logs data protection
policies. Restores environment.py (sagemaker-train and the sagemaker-core
mirror) and its tests to their original state.
* Fix role issue in mtrl integ tests * Fix role issue in mtrl integ tests * Fix: Telemetry INFO logging to print only once per process * Fix: Telemetry INFO logging to print only once per process * Fix: Telemetry INFO logging to print only once per process * Fix: Telemetry INFO logging to print only once per process --------- Co-authored-by: Roja Reddy Sareddy <rsareddy@amazon.com>
…lder (aws#6134) The 3rd cell of v3-examples/model-customization-examples/ mtrl_finetuning_example_notebook_v3_prod.ipynb set S3_TRAINING_DATA using an f-string that referenced an undefined STAGE variable and an internal `-rftjob-input` bucket pattern left over from when MTRL was tested in gamma/prod. Executing the cell raised `NameError: name 'STAGE' is not defined`. Replace it with a customer-facing placeholder S3 URI, consistent with the sibling sft_finetuning notebook, so users (and the notebook test engine) can substitute their own dataset path. Mirrors the fix made to the same notebook in aws/amazon-sagemaker-examples. Co-authored-by: hrehard <hrehard@amazon.com>
…eploy (aws#6101) * fix(serve): remove IC data-source collapse hack from recommendation deploy Inference Components now support AdditionalModelDataSources natively (kernel tuning / speculative decoding channels), so _deploy_recommendation no longer needs to collapse an optimized recommendation's base_model + draft channels into a single primary ModelDataSource. Deploy the recommendation's ModelPackage directly and let the hosting stack resolve the channels. Removes the base_model-promotion / OPTION_SPECULATIVE_DRAFT_MODEL rewiring block and its now-unused shape imports (AdditionalModelDataSource, ModelDataSource, S3ModelDataSource, SPECULATIVE_DRAFT_MODEL). Updates the speculative-decoding unit tests to assert the ModelPackage pass-through. * test(serve): integ test deploying an SD/KT model as an Inference Component Adds an end-to-end integration test that builds a model carrying speculative-decoding / kernel-tuning AdditionalModelDataSources (base + draft channels) and deploys it as an Inference Component via ModelBuilder.deploy(inference_config=ResourceRequirements(...)). Asserts the Inference Component reaches InService and that the deployed model still carries the additional sources (guards against a silent client-side collapse). Marked slow_test + gpu_intensive. * test(serve): fix SD/KT IC integ test build path and trim docstring The test built the IC model via ModelBuilder(model_path=<s3_uri>), which raises 'Cannot detect required model or inference spec' — a raw S3 path is not a buildable model spec. Build via ModelBuilder.from_jumpstart_config so the container/framework resolve, then attach additional_model_data_sources (carried onto the model through _prepare_container_def_base) before deploying as an Inference Component. Also switch the IC instance to g4dn.xlarge (a GPU is needed for the vLLM/LMI container, not for the 0.6B size) and trim the module docstring. * test(serve): poll IC to terminal state before asserting + harden teardown deploy(wait=True) waits for the endpoint, but the Inference Component is created with wait=False, so the IC can still be Creating when deploy() returns. Add _wait_for_ic_terminal to poll the IC to InService/Failed before the status assertion, and wait it out of Creating before teardown (an IC in Creating cannot be deleted, which would strand its GPU endpoint).
…ir (aws#6147) * fix(serve): create local model_path dir before using it as download dir ModelBuilder.build() assigned the default model_path (/tmp/sagemaker/model-builder/<uuid>) to settings._local_download_dir without creating it on disk, so repack_model()'s _tmpdir() validation raised "Inputted directory ... does not exist" for source_code repack builds. Only use model_path as the local download dir when it is a local path, creating it first; skip s3:// URIs. * add unit tests
…/deployment/evaluation subfolders
get_jumpstart_configs accepted no tolerate_vulnerable_model or tolerate_deprecated_model argument. It called verify_model_region_and_return_specs without them, so the callee fell back to its False defaults and re-ran the model gate. A caller that had asked to tolerate a flagged model still got VulnerableJumpStartModelError or DeprecatedJumpStartModelError, which made both flags unusable for that model. Add both parameters, default them to False to keep current behavior for existing callers, and forward them to verify_model_region_and_return_specs. Pass them from ModelBuilder._ensure_metadata_configs, which resolves the same configs lazily and had no way to opt out of the gate. --- X-AI-Prompt: Can you fix the dropped JumpStart tolerance flags in v3 too? X-AI-Tool: claude-code
…rces, show_metrics, and stream_logs
**Description** **Testing Done**
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run
git merge masterto resolve merge conflict