Skip to content

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

15 Commits

Folders and files

Repository files navigation

DeepSpeed Benchmark Harness

This repository benchmarks DeepSpeed releases against a configurable workload matrix and writes timing, throughput, memory, and loss reports for each run and comparison. The harness is intended for DeepSpeed benchmarking across accelerator environments, not only CUDA devices.

This repository currently ships a release-benchmark workflow. The example benchmark contract in configs/llama3_8b_release_matrix.yaml uses:

  • model: meta-llama/Meta-Llama-3-8B
  • dataset: wikitext / wikitext-103-raw-v1
  • dtype: bf16
  • ZeRO stages: 0, 1, 2, 3
  • batch sizes: 1, 2, 4
  • sequence lengths: 1024, 2048, 4096
  • deterministic mode enabled with seed=42

The repository includes the training workload implementation used by the benchmark scripts.

Current Charts

The repository checks in compact benchmark history and generated charts for quick regression review. The current charts below are generated from benchmarks/history.csv for llama3_8b_regression_zero123 on 8 GPUs.

The history now includes DeepSpeed v0.19.0 through v0.19.5, measured with benchmark source 667a855ad7edccb1da97c98366a168a2c6dce394. All 54 v0.19.x rows completed on one node with eight H100 GPUs, and every row passed the publication gate with exactly 40 finite loss points and successful rank summaries for global ranks 0 through 7. Across the 45 adjacent v0.19.x comparisons, no throughput, iteration-time, or max-rank peak-memory change reached 5%. The largest absolute throughput change was -3.97% from v0.19.2 to v0.19.3 for ZeRO-3 at sequence length 1024.

These values are descriptive rather than statistically significant: each v0.19.x release has one non-replicated run, and the checked-in v0.18.9 rows came from an earlier environment. The v0.19.x matrix used Transformers 5.9.0. Transformers 5.8.x is intentionally unsupported because its ZeRO-3 config-only BF16 Linear initializer operated on module.weight.float() without copying the initialized values back, leaving parameters nonfinite before training. The official 5.9.0 wheel used for the matrix has SHA-256 1d19509bcff7028ebc6b277d71caa712e8353778463d38764237d14b42b52788.

Throughput

Global throughput by release, ZeRO-1

Global throughput by release, ZeRO-2

Global throughput by release, ZeRO-3

Memory

The memory chart uses one fixed condition so the release comparison is easy to read: batch_size=1, seq_length=1024, with separate lines for ZeRO-1/2/3.

Peak memory by release for batch size 1 and sequence length 1024

Prerequisites

  • Linux with a DeepSpeed-supported accelerator environment.
  • The current built-in release workflow and example matrix target an NVIDIA GPU setup and expect 8 visible devices; lower benchmark.gpus, sequence lengths, or batch sizes if your machine is smaller.
  • A Python environment with a DeepSpeed-compatible PyTorch stack plus accelerate, transformers==5.9.0, datasets, and wandb.
  • Hugging Face access to the model config/tokenizer used by the benchmark.
  • A writable output path with enough free space for mirrors, release checkouts, logs, and plots.

Usage

Clone the repository and run the scripts directly from the repo root. The scripts add src/ to sys.path, so the harness itself does not need to be installed as a package.

Install the harness dependencies, including the exact supported Transformers version, from the checked-in project metadata:

python -m pip install -e .

For local development and tests, also install pytest:

python -m pip install pytest

The harness writes heavy artifacts under the repo's ignored .artifacts/ directory by default. For real runs, set BENCH_ROOT to an external output root on a disk with enough space.

If --output-root is omitted, the tool falls back to <repo>/.artifacts/.

You can also set DS_BENCH_OUTPUT_ROOT=/path/to/output and omit --output-root on script invocations.

Runtime Environment

Each release is installed into an isolated runtime environment at <output-root>/envs/runtime.

There are two supported bootstrap modes:

  • Default: create a virtual environment from the current Python with --system-site-packages.
  • Optional: clone an existing conda environment with --seed-env <name-or-prefix>.

The tool installs Transformers 5.9.0 and the DeepSpeed release under test into that runtime environment, then verifies the installed Transformers version before running. It does not build an accelerator runtime or PyTorch stack from scratch.

Configuration

Benchmark matrices are defined in YAML under configs/. The current config schema includes:

  • model_name: Hugging Face model identifier used by the benchmark workload
  • load_weights: load pretrained weights with from_pretrained() instead of constructing from config
  • dataset_name and optional dataset_variant: dataset selection passed to datasets.load_dataset()
  • dtype: bf16 or fp16
  • releases.seed: release tags used by run-batch when --releases is omitted
  • zero_stages, seq_lengths, batch_sizes: matrix dimensions expanded into benchmark conditions
  • gpus: expected visible device count for the current built-in workflow
  • port_base: starting port for accelerate process-group setup

The example config is intentionally conservative about scope. The harness is not hardcoded to only wikitext or only one dataset variant, although the current checked-in preflight and telemetry path is still oriented around the NVIDIA GPU example workflow.

Source Repository

By default, the harness seeds its local mirror from the official DeepSpeed GitHub repository:

https://github.com/deepspeedai/DeepSpeed.git

To benchmark a fork or a local checkout instead, pass --deepspeed-repo <url-or-path>.

Output Layout

Under --output-root, the scripts create:

  • sources/DeepSpeed.git: mirror used to fetch tags
  • checkouts/<tag>/DeepSpeed: release checkout for a tag
  • envs/runtime: isolated runtime environment
  • cache/: Hugging Face, pip, and temporary caches
  • runs/<tag>/run-<timestamp>/: logs, metrics, summaries, and plots
  • runs/comparisons/<baseline>__vs__<candidate>/: analysis reports and loss overlays
  • state/progress.json: resumable progress state

Checked-in benchmark history is kept separately under benchmarks/. Raw logs, condition artifacts, and heavyweight caches stay under --output-root.

Scripts

Script Required arguments Purpose Typical use
scripts/run_release.py --release, --config Install the requested DeepSpeed tag into the runtime environment, run preflight checks, execute the benchmark matrix, resume incomplete runs, and write run artifacts and summaries. Benchmark a single release end to end.
scripts/run_batch.py --config Run run_release.py for multiple releases, either from --releases or releases.seed in the config file. Benchmark a release history with one command.
scripts/analyze.py --baseline, --candidate Compare two completed release runs and write a comparison report. Analyze differences after both releases have been benchmarked.
scripts/export_history.py --run-dir ... or --output-root + --release ... Flatten completed run artifacts into a compact CSV for long-term history tracking. Check a small, chart-ready benchmark history into git.
scripts/plot_history.py none Read benchmarks/history.csv and render release-to-release throughput and memory charts. Regenerate checked-in charts after adding runs to history.

Run a release benchmark:

python scripts/run_release.py \
  --release v0.18.7 \
  --config configs/llama3_8b_release_matrix.yaml \
  --output-root "$BENCH_ROOT"

Resume an interrupted run without redoing completed conditions:

python scripts/run_release.py \
  --release v0.18.7 \
  --config configs/llama3_8b_release_matrix.yaml \
  --output-root "$BENCH_ROOT"

Benchmark a local DeepSpeed checkout or fork:

python scripts/run_release.py \
  --release v0.18.7 \
  --config configs/llama3_8b_release_matrix.yaml \
  --output-root "$BENCH_ROOT" \
  --deepspeed-repo ../DeepSpeed

Run a batch of releases:

python scripts/run_batch.py \
  --releases v0.18.5 v0.18.6 v0.18.7 \
  --config configs/llama3_8b_release_matrix.yaml \
  --output-root "$BENCH_ROOT"

Analyze two complete releases:

python scripts/analyze.py \
  --baseline v0.18.6 \
  --candidate v0.18.7 \
  --output-root "$BENCH_ROOT"

Export a flat history CSV from existing run artifacts:

python scripts/export_history.py \
  --config configs/llama3_8b_regression_zero123.yaml \
  --run-dir "$BENCH_ROOT/runs/v0.18.6/run-20260330T064726Z" \
  --run-dir "$BENCH_ROOT/runs/v0.18.7/run-20260330T070928Z" \
  --run-dir "$BENCH_ROOT/runs/v0.18.8/run-20260330T072401Z" \
  --config-name llama3_8b_regression_zero123 \
  --output benchmarks/history.csv

Generate checked-in throughput and memory charts from the flat history:

python scripts/plot_history.py \
  --history-csv benchmarks/history.csv \
  --output-dir benchmarks/charts \
  --config-name llama3_8b_regression_zero123 \
  --gpus 8

History And Charts

The repository is meant to store compact benchmark history, not raw benchmark artifacts.

  • Keep raw run data under an external --output-root; do not check raw artifacts into this repository
  • Export one flat row per release + condition into benchmarks/history.csv
  • Generate charts from that CSV into benchmarks/charts/
  • If the history contains multiple GPU counts for the same config, pass --gpus to scripts/plot_history.py so the charts only use one machine shape at a time

The history CSV is generated from run artifacts, not edited by hand. Successful rows are exported only after the configured loss-point count is finite and complete and the exact expected global-rank set has successful summaries. Machine-local artifact paths are normalized to logical paths under runs/ before publication.

Notes

  • Reported iteration times are deterministic-mode timings. They are intended for controlled comparisons, not peak-throughput marketing numbers.
  • Reported throughput in benchmarks/history.csv and the generated charts is derived from deterministic iteration time, batch size, sequence length, and visible GPU count. It is intended for regression tracking, not peak-throughput marketing numbers.
  • The sample config uses load_weights=false, so it measures release behavior from from_config() initialization rather than from downloaded model weights.
  • If you want to keep the progress file somewhere else, pass --progress-file state/progress.json or set DS_BENCH_PROGRESS_FILE.

Validation

Lightweight local validation:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 python -m pytest -q
python scripts/run_release.py --help
python scripts/run_batch.py --help
python scripts/analyze.py --help
python scripts/export_history.py --help
python scripts/plot_history.py --help

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages