This repository benchmarks DeepSpeed releases against a configurable workload matrix and writes timing, throughput, memory, and loss reports for each run and comparison. The harness is intended for DeepSpeed benchmarking across accelerator environments, not only CUDA devices.
This repository currently ships a release-benchmark workflow. The example benchmark contract in configs/llama3_8b_release_matrix.yaml uses:
- model:
meta-llama/Meta-Llama-3-8B - dataset:
wikitext/wikitext-103-raw-v1 - dtype:
bf16 - ZeRO stages:
0, 1, 2, 3 - batch sizes:
1, 2, 4 - sequence lengths:
1024, 2048, 4096 - deterministic mode enabled with
seed=42
The repository includes the training workload implementation used by the benchmark scripts.
The repository checks in compact benchmark history and generated charts for quick regression review. The current charts below are generated from benchmarks/history.csv for llama3_8b_regression_zero123 on 8 GPUs.
The history now includes DeepSpeed v0.19.0 through v0.19.5, measured with benchmark source 667a855ad7edccb1da97c98366a168a2c6dce394. All 54 v0.19.x rows completed on one node with eight H100 GPUs, and every row passed the publication gate with exactly 40 finite loss points and successful rank summaries for global ranks 0 through 7. Across the 45 adjacent v0.19.x comparisons, no throughput, iteration-time, or max-rank peak-memory change reached 5%. The largest absolute throughput change was -3.97% from v0.19.2 to v0.19.3 for ZeRO-3 at sequence length 1024.
These values are descriptive rather than statistically significant: each v0.19.x release has one non-replicated run, and the checked-in v0.18.9 rows came from an earlier environment. The v0.19.x matrix used Transformers 5.9.0. Transformers 5.8.x is intentionally unsupported because its ZeRO-3 config-only BF16 Linear initializer operated on module.weight.float() without copying the initialized values back, leaving parameters nonfinite before training. The official 5.9.0 wheel used for the matrix has SHA-256 1d19509bcff7028ebc6b277d71caa712e8353778463d38764237d14b42b52788.
The memory chart uses one fixed condition so the release comparison is easy to read: batch_size=1, seq_length=1024, with separate lines for ZeRO-1/2/3.
- Linux with a DeepSpeed-supported accelerator environment.
- The current built-in release workflow and example matrix target an NVIDIA GPU setup and expect 8 visible devices; lower
benchmark.gpus, sequence lengths, or batch sizes if your machine is smaller. - A Python environment with a DeepSpeed-compatible PyTorch stack plus
accelerate,transformers==5.9.0,datasets, andwandb. - Hugging Face access to the model config/tokenizer used by the benchmark.
- A writable output path with enough free space for mirrors, release checkouts, logs, and plots.
Clone the repository and run the scripts directly from the repo root. The scripts add src/ to sys.path, so the harness itself does not need to be installed as a package.
Install the harness dependencies, including the exact supported Transformers version, from the checked-in project metadata:
python -m pip install -e .For local development and tests, also install pytest:
python -m pip install pytestThe harness writes heavy artifacts under the repo's ignored .artifacts/ directory by default. For real runs, set BENCH_ROOT to an external output root on a disk with enough space.
If --output-root is omitted, the tool falls back to <repo>/.artifacts/.
You can also set DS_BENCH_OUTPUT_ROOT=/path/to/output and omit --output-root on script invocations.
Each release is installed into an isolated runtime environment at <output-root>/envs/runtime.
There are two supported bootstrap modes:
- Default: create a virtual environment from the current Python with
--system-site-packages. - Optional: clone an existing conda environment with
--seed-env <name-or-prefix>.
The tool installs Transformers 5.9.0 and the DeepSpeed release under test into that runtime environment, then verifies the installed Transformers version before running. It does not build an accelerator runtime or PyTorch stack from scratch.
Benchmark matrices are defined in YAML under configs/. The current config schema includes:
model_name: Hugging Face model identifier used by the benchmark workloadload_weights: load pretrained weights withfrom_pretrained()instead of constructing from configdataset_nameand optionaldataset_variant: dataset selection passed todatasets.load_dataset()dtype:bf16orfp16releases.seed: release tags used byrun-batchwhen--releasesis omittedzero_stages,seq_lengths,batch_sizes: matrix dimensions expanded into benchmark conditionsgpus: expected visible device count for the current built-in workflowport_base: starting port foraccelerateprocess-group setup
The example config is intentionally conservative about scope. The harness is not hardcoded to only wikitext or only one dataset variant, although the current checked-in preflight and telemetry path is still oriented around the NVIDIA GPU example workflow.
By default, the harness seeds its local mirror from the official DeepSpeed GitHub repository:
https://github.com/deepspeedai/DeepSpeed.git
To benchmark a fork or a local checkout instead, pass --deepspeed-repo <url-or-path>.
Under --output-root, the scripts create:
sources/DeepSpeed.git: mirror used to fetch tagscheckouts/<tag>/DeepSpeed: release checkout for a tagenvs/runtime: isolated runtime environmentcache/: Hugging Face, pip, and temporary cachesruns/<tag>/run-<timestamp>/: logs, metrics, summaries, and plotsruns/comparisons/<baseline>__vs__<candidate>/: analysis reports and loss overlaysstate/progress.json: resumable progress state
Checked-in benchmark history is kept separately under benchmarks/. Raw logs, condition artifacts, and heavyweight caches stay under --output-root.
| Script | Required arguments | Purpose | Typical use |
|---|---|---|---|
scripts/run_release.py |
--release, --config |
Install the requested DeepSpeed tag into the runtime environment, run preflight checks, execute the benchmark matrix, resume incomplete runs, and write run artifacts and summaries. | Benchmark a single release end to end. |
scripts/run_batch.py |
--config |
Run run_release.py for multiple releases, either from --releases or releases.seed in the config file. |
Benchmark a release history with one command. |
scripts/analyze.py |
--baseline, --candidate |
Compare two completed release runs and write a comparison report. | Analyze differences after both releases have been benchmarked. |
scripts/export_history.py |
--run-dir ... or --output-root + --release ... |
Flatten completed run artifacts into a compact CSV for long-term history tracking. | Check a small, chart-ready benchmark history into git. |
scripts/plot_history.py |
none | Read benchmarks/history.csv and render release-to-release throughput and memory charts. |
Regenerate checked-in charts after adding runs to history. |
Run a release benchmark:
python scripts/run_release.py \
--release v0.18.7 \
--config configs/llama3_8b_release_matrix.yaml \
--output-root "$BENCH_ROOT"Resume an interrupted run without redoing completed conditions:
python scripts/run_release.py \
--release v0.18.7 \
--config configs/llama3_8b_release_matrix.yaml \
--output-root "$BENCH_ROOT"Benchmark a local DeepSpeed checkout or fork:
python scripts/run_release.py \
--release v0.18.7 \
--config configs/llama3_8b_release_matrix.yaml \
--output-root "$BENCH_ROOT" \
--deepspeed-repo ../DeepSpeedRun a batch of releases:
python scripts/run_batch.py \
--releases v0.18.5 v0.18.6 v0.18.7 \
--config configs/llama3_8b_release_matrix.yaml \
--output-root "$BENCH_ROOT"Analyze two complete releases:
python scripts/analyze.py \
--baseline v0.18.6 \
--candidate v0.18.7 \
--output-root "$BENCH_ROOT"Export a flat history CSV from existing run artifacts:
python scripts/export_history.py \
--config configs/llama3_8b_regression_zero123.yaml \
--run-dir "$BENCH_ROOT/runs/v0.18.6/run-20260330T064726Z" \
--run-dir "$BENCH_ROOT/runs/v0.18.7/run-20260330T070928Z" \
--run-dir "$BENCH_ROOT/runs/v0.18.8/run-20260330T072401Z" \
--config-name llama3_8b_regression_zero123 \
--output benchmarks/history.csvGenerate checked-in throughput and memory charts from the flat history:
python scripts/plot_history.py \
--history-csv benchmarks/history.csv \
--output-dir benchmarks/charts \
--config-name llama3_8b_regression_zero123 \
--gpus 8The repository is meant to store compact benchmark history, not raw benchmark artifacts.
- Keep raw run data under an external
--output-root; do not check raw artifacts into this repository - Export one flat row per
release + conditionintobenchmarks/history.csv - Generate charts from that CSV into
benchmarks/charts/ - If the history contains multiple GPU counts for the same config, pass
--gpustoscripts/plot_history.pyso the charts only use one machine shape at a time
The history CSV is generated from run artifacts, not edited by hand. Successful rows are exported only after the configured loss-point count is finite and complete and the exact expected global-rank set has successful summaries. Machine-local artifact paths are normalized to logical paths under runs/ before publication.
- Reported iteration times are deterministic-mode timings. They are intended for controlled comparisons, not peak-throughput marketing numbers.
- Reported throughput in
benchmarks/history.csvand the generated charts is derived from deterministic iteration time, batch size, sequence length, and visible GPU count. It is intended for regression tracking, not peak-throughput marketing numbers. - The sample config uses
load_weights=false, so it measures release behavior fromfrom_config()initialization rather than from downloaded model weights. - If you want to keep the progress file somewhere else, pass
--progress-file state/progress.jsonor setDS_BENCH_PROGRESS_FILE.
Lightweight local validation:
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -q
python scripts/run_release.py --help
python scripts/run_batch.py --help
python scripts/analyze.py --help
python scripts/export_history.py --help
python scripts/plot_history.py --help


