feat(serve): list/filter jobs, deploy from recommendation row, compare benchmarks, DataFrame views - #6148
Open
ZealSV wants to merge 1 commit into
Open
feat(serve): list/filter jobs, deploy from recommendation row, compare benchmarks, DataFrame views#6148ZealSV wants to merge 1 commit into
ZealSV wants to merge 1 commit into
Conversation
Lokiiiiii
suggested changes
Aug 4, 2026
ZealSV
force-pushed
the
pysdk-benchmark-rec-enhancements
branch
from
August 5, 2026 00:57
d521c45 to
7d0b974
Compare
Lokiiiiii
suggested changes
Aug 5, 2026
…e benchmarks, DataFrame views Adds inference-recommender usability features to sagemaker.serve: - list_benchmarks(endpoint=...) / list_recommendations(model=..., model_package=...): the ListAI* APIs cannot filter by endpoint/model server-side (those fields are on Describe, not the list summary), so filtering is client-side. Hydration is the resource iterator's job (no redundant Describe); max_results caps matches returned and max_scan caps candidates described, so a rarely-matching filter cannot fan out across the whole account. A candidate that fails to Describe is skipped and logged (not fatal), and sagemaker_session accepts either a boto3 Session or a sagemaker Session (unwrapped to its boto_session). - ModelBuilder.deploy(recommendation=<row>): accept a recommendation row (mb.recommendations.best or [i]) and pass its underlying shape straight through so the exact row is deployed — not a positional index, which can go stale (the row list is re-read at deploy time) or point into a different job. A non-row value raises TypeError rather than silently defaulting to the top row. Additive and back-compat; existing index/spec_name kwargs still work. - compare_benchmarks(*results): N-way comparison, first result = baseline, one row per metric x one column per run + a signed delta% column. The delta is oriented so + is better only for metrics with a known direction (throughput higher-better, latency/duration lower-better); directionless metrics (sequence lengths, token counts, HTTP counters) get no signed delta. Rejects duplicate run names and names colliding with the reserved "unit" column. - to_dataframe() on every tabular surface (customer-requested DataFrame view): BenchmarkMetrics, BenchmarkResult, BenchmarkComparison, and both recommendation views. Each returns a pandas DataFrame mirroring its printed table but carrying the extra stats (min/max/p95/stddev) the width-limited text tables drop, with numeric columns coerced to float64 so a metric absent from every row is NaN (sortable), not object. pandas is imported lazily so result.py stays dependency-light. __str__ and to_dataframe() share ordering/row-building helpers so the printed table and the frame never drift. Unit: full recommender suite 223 pass. Integ: test_ai_inference_recommender_enhancements_integration.py chains rec -> deploy(rows.best) InService -> 2 benchmarks -> compare_benchmarks; the no-GPU plumbing test caps max_scan so it stays cheap on PR checks.
ZealSV
force-pushed
the
pysdk-benchmark-rec-enhancements
branch
from
August 5, 2026 19:54
7d0b974 to
a305a7f
Compare
Lokiiiiii
approved these changes
Aug 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds inference-recommender usability features to sagemaker.serve:
list_benchmarks(endpoint=...) / list_recommendations(model=..., model_package=...): the ListAI* APIs cannot filter by endpoint/model server-side (those fields are on Describe, not the list summary), so filtering is client-side (list -> describe -> match) bounded by max_results. Chosen because it is the only option the boto APIs allow, and a bounded describe fan-out keeps a broad list from being unbounded.
ModelBuilder.deploy(recommendation=): accept a recommendation row (mb.recommendations.best or [i]) and resolve it to the existing spec-name/index selection. Chosen as an additive, back-compat param so callers deploy the best row without hand-copying a magic index; existing index/spec_name kwargs still work.
compare_benchmarks(*results): N-way comparison, first result = baseline, one row per metric x one column per run + a signed delta% column oriented so + is always better. Chosen N-way (not strictly 2-way) because it is barely more code and the delta vs a baseline is the whole point of a comparison utility.
to_dataframe() on every tabular surface (customer-requested DataFrame view): BenchmarkMetrics, BenchmarkResult, BenchmarkComparison, and both recommendation views (_RecommendationView, _RecommendationsView). Each returns a pandas DataFrame mirroring its printed table, but carrying the extra stats (min/max/p95/stddev) the width-limited text tables drop, and keeping numeric values native (deltas numeric, NaN where undefined) instead of preformatted strings. pandas is imported lazily via _require_pandas() so result.py stays dependency-light (its printed tables are stdlib-only); it is present transitively via sagemaker-core in any real install. str and to_dataframe() share ordering/row-building helpers (_ordered_metric_pairs, _row_records, _delta_value) so the printed table and the frame never drift. BenchmarkResult.to_dataframe() raises on a search/sweep result (no single profile).
Unit: 45 new tests (11 listing, 5 deploy-row, 12 compare, 17 to_dataframe); full recommender suite 199 pass. Integ: test_ai_inference_recommender_enhancements_integration.py chains rec -> deploy(rows.best) InService -> 2 benchmarks -> compare_benchmarks; verified live (1 passed, 70 min) plus a no-GPU listing-plumbing test.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.