A self-hosted, on-demand package mirror for PyPI, npm, crates.io, Debian, Ubuntu, and Alpine, with a React/shadcn admin and user dashboards embedded in a single Rust service. Speculum listens on HTTP; place your own TLS-terminating reverse proxy in front of it.
- Run
docker compose pullfollowed bydocker compose up -d --wait. Compose uses the prebuilt Linux AMD64 imageghcr.io/amirhmzz/speculum:latest; no registry login or credential environment variables are needed once the package is public. - Read the generated initial password with
docker compose logs speculum. Look forSPECULUM_INITIAL_ADMIN_PASSWORD=.... It is printed once on a fresh database, even whenRUST_LOGfilters other startup messages. - Connect your reverse proxy to the Compose network and route it to
http://speculum:8080. Open/adminon your public URL and sign in with the username admin and the generated password. Choose a new password before accessing the dashboard. - In Settings, set the public base URL to the HTTPS origin your clients use.
- Configure each registry's proxy, authentication, and cache lifetime as needed.
- Add accounts under Mirror users. Copy each generated temporary password
and share it with that user. They sign in at
/account, change their password, and create package tokens. Client configuration is available in both dashboards.
The container listens on 0.0.0.0:8080, accessible within its Docker network.
Compose does not publish any ports to the host. Attach your reverse proxy to the
same Docker network and forward to http://speculum:8080, preserving the request
path. Preserve all registry, distro, and /api/ prefixes. Allow enough request time
for a cold large artifact: it is downloaded completely before being served.
The named speculum-data volume stores configuration, users, sessions, statistics,
metadata, artifacts, and secret.key. Runtime settings are edited through the
dashboard and take effect without restarting. The administrator username is always
admin. Restarts preserve the chosen password and never print it. Bootstrap
passwords are stored only as hashes; use password recovery if the initial log is lost.
From the same directory and Compose project used for installation, update manually:
docker compose pull
docker compose up -d --waitUpdates reuse the existing data volume. Back up the complete volume before an upgrade, especially when the release changes the database schema (see Operations and recovery). Speculum does not migrate existing schemas automatically.
The latest tag follows successful stable release-tag builds. To pin a version or
digest, set SPECULUM_IMAGE in the .env file beside compose.yaml, then run the
commands above:
SPECULUM_IMAGE=ghcr.io/amirhmzz/speculum:0.2.1
# Or use the digest shown in the publishing workflow summary:
# SPECULUM_IMAGE=ghcr.io/amirhmzz/speculum@sha256:<published-digest>Use ghcr.io/amirhmzz/speculum:main to follow tested development builds. To build
your checkout locally instead, explicitly include the development override on
every Compose command (including logs, restart, and recovery):
docker compose -f compose.yaml -f compose.build.yaml up --build -d --waitThis builds speculum:local and preserves the same service settings and data
volume. The standard Compose file never builds from source.
For local access without a reverse proxy, also include compose.local.yaml:
docker compose -f compose.yaml -f compose.build.yaml -f compose.local.yaml up --build -d --waitOpen http://127.0.0.1:8080/admin. This publishes the port only on localhost.
Include the same override files in subsequent Compose commands. To use the
published image locally, omit -f compose.build.yaml and --build.
GitHub Actions validates pull requests targeting main, pushes to main, and
version-tag pushes. It builds the dashboard using its frozen Bun lockfile, runs
Rust formatting/Clippy/tests, and tests real uv/npm/Bun/Cargo installs against
fixture registries, including offline cache use. Rust and Bun versions are read
from the Dockerfile. A separate job builds the Linux AMD64 image and smoke-tests
its health, embedded dashboards, account setup, and restart persistence using a
unique temporary Compose project. Test storage is removed afterward.
Run the publishing-boundary and container checks locally with:
python -m unittest discover -s tests -p 'test_ci_*.py'
docker compose -f compose.yaml -f compose.build.yaml build
python tests/docker_ci.py --image speculum:localPublishing runs only after every check passes on a main or release-tag push. It loads and verifies the tested image from a workflow artifact rather than rebuilding it. Pull requests never publish. Published image tags and the registry digest appear in the publishing job's summary:
| Trigger | Tags |
|---|---|
Push to main |
main, sha-<full-commit> |
Push v0.2.1 |
0.2.1, 0.2, latest, sha-<full-commit> |
New release tags must use vX.Y.Z, without prerelease suffixes or leading zeros.
Existing tags are unchanged. After updating package versions/changelog as needed
and merging the intended release commit to main, create a new, unused tag:
git tag v0.2.1
git push origin v0.2.1Replace the example with your intended version. The workflow does not change
package versions, create GitHub releases, or update running servers. Each successful
release-tag push updates its minor alias and latest, including a backport; pin
an exact version or digest when that behavior is undesirable. The SHA tag identifies
the source commit; main and release builds of that commit may have different
version labels. Use a digest to identify the exact image.
Enable GitHub Actions for the repository and allow the pinned actions used by
the workflow. The publishing job uses the built-in GITHUB_TOKEN with
packages: write; no personal access token or repository secret is required.
The OCI source label links the image to this repository. If a package with this
name already exists, ensure this repository has Actions write access to it.
GHCR initially creates packages as private, even for public repositories. After the first successful publication, open your GitHub profile's Packages, select speculum, open Package settings, and change its visibility to Public. This is a one-time package-owner setting; the workflow does not change visibility. See GitHub's Container registry documentation.
Verify an anonymous pull using an empty Docker credential configuration:
docker --config /tmp/speculum-ghcr-anonymous pull ghcr.io/amirhmzz/speculum:mainUse a fresh, empty directory for --config (on Windows, use a temporary Windows
path). latest becomes available after the first successful vX.Y.Z build; a
main-only first publication provides main and SHA tags. Until a stable image is
published, set SPECULUM_IMAGE=ghcr.io/amirhmzz/speculum:main or use the local-build
override.
All registries initially require authentication. Mirror credentials are separate from dashboard passwords. Disabling registry authentication opens only that registry; the admin API always requires an admin session. Mirror-user passwords authenticate only the user dashboard. All protected package downloads use tokens, including Python Basic-auth passwords. Disabling a user immediately blocks dashboard access and all their tokens; deletion also removes their tokens, sessions, and personal statistics. Admins can still issue tokens for users. Password resets generate a new temporary password, require another password change, and sign out user sessions while preserving package tokens.
Both dashboard roles sign in at /login using one form. Speculum detects the
account role and redirects administrators to /admin and mirror users to
/account, including when a password change is required.
Both dashboards show the same global overview, package inventory, storage, and
lifetime traffic statistics, including package versions and cached file details.
Users can browse this shared data while managing only their own tokens and account.
The /api/user/ APIs provide login/session/logout, password changes, tokens,
global summary, paginated packages, package details, and public client configuration.
Cache clearing and package, version, or artifact deletion remain admin-only. Admin
sessions and user sessions are separate; users cannot access /api/admin/.
The Tokens table shows when each token last authenticated a mirror request,
including metadata and HEAD requests. Unused tokens show Never. Invalid
credentials and disabled users do not update this timestamp.
uv — use a named index in pyproject.toml, with credentials in environment
variables:
[[tool.uv.index]]
name = "speculum"
url = "https://mirror.example.com/pypi/simple/"
default = trueSet UV_INDEX_SPECULUM_USERNAME to the mirror username and
UV_INDEX_SPECULUM_PASSWORD to a token belonging to that user. Standard pip
clients use the same username and token through HTTP Basic authentication
(for example through their credential store or .netrc). The token occupies
the client's password field; Speculum does not store or accept mirror-user
passwords. Renaming a user requires updating that username in Python clients.
HTML and JSON Simple API responses, metadata sidecars, and byte ranges
are supported. The index root lists discovered projects; named projects are
resolved lazily without downloading the entire upstream project catalogue.
npm — put the following in .npmrc and set SPECULUM_TOKEN:
registry=https://mirror.example.com/npm/
//mirror.example.com/npm/:_authToken=${SPECULUM_TOKEN}Scoped packages, exact versions, tags, full/abbreviated metadata, and npm audit
requests are supported. Conventional tarball paths (/npm/name/-/name-version.tgz
and /npm/@scope/name/-/name-version.tgz) resolve on demand and share the cache
with metadata-generated download URLs, including requests from existing Bun
lockfiles. npm login/publishing are not supported; issue tokens in
the dashboard. Existing lockfiles that contain direct upstream tarball URLs can
bypass the mirror: regenerate those lockfiles using the mirror registry.
Mirror roots also accept URLs without a trailing slash. /npm and
/pypi/simple redirect to their canonical roots; /cargo/index redirects to
the sparse registry's config.json. Authentication and registry settings
apply to these aliases.
Bun — use the same npm endpoint. Put this in bunfig.toml at the project
root, set SPECULUM_TOKEN to a mirror-user token, and run bun install:
[install]
registry = { url = "https://mirror.example.com/npm/", token = "$SPECULUM_TOKEN" }Bun also reads the .npmrc configuration above. When npm authentication is
disabled, use registry = "https://mirror.example.com/npm/" without a token.
See Bun registry configuration.
Deno — put this .npmrc in the project root and set SPECULUM_TOKEN to a
mirror-user token:
registry=https://mirror.example.com/npm/
//mirror.example.com/npm/:_authToken=${SPECULUM_TOKEN}Declare npm dependencies in deno.json, for example:
{
"imports": {
"chalk": "npm:chalk@5"
}
}Run deno install, then import the configured alias in your code (for example
import chalk from "chalk"). Existing package.json dependencies also use the
configured npm registry. When npm authentication is disabled, omit the
_authToken line. Only Deno's npm dependencies use Speculum; jsr: packages and
HTTP imports use their original sources. See
Deno npm registry configuration.
Cargo — put this in .cargo/config.toml and set
CARGO_REGISTRIES_SPECULUM_TOKEN:
[source.crates-io]
replace-with = "speculum"
[registries.speculum]
index = "sparse+https://mirror.example.com/cargo/index/"
credential-provider = "cargo:token"Cargo's config.json is public discovery data containing the mirror download
URL and authentication flag. Index entries and crate files are protected when
Cargo authentication is enabled. Only the sparse index is supported. Git/path
dependencies and package publishing are outside the mirror's scope.
The archive roots are /debian/, /debian-security/, /ubuntu/,
/ubuntu-security/, and /ubuntu-ports/. Upstream URLs, proxies, authentication,
freshness, and retention are configured independently in Settings. Ubuntu ports
serves ARM64 and the other ports architectures; use /ubuntu/ for AMD64/i386.
Suites, components, and architectures are selected by the client.
Replace matching existing sources with /etc/apt/sources.list.d/speculum.sources.
For Debian, replace <release> with your installed release codename:
Types: deb
URIs: https://mirror.example.com/debian/
Suites: <release> <release>-updates <release>-backports
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
Types: deb
URIs: https://mirror.example.com/debian-security/
Suites: <release>-security
Components: main contrib non-free non-free-firmware
Signed-By: /usr/share/keyrings/debian-archive-keyring.gpg
For Ubuntu, use /ubuntu/ and /ubuntu-security/, components
main restricted universe multiverse, and
/usr/share/keyrings/ubuntu-archive-keyring.gpg. For Ubuntu ports, use
/ubuntu-ports/ for all suites, including <release>-security.
Use Types: deb deb-src if source packages are also needed.
When authentication is enabled, save the following in
/etc/apt/auth.conf.d/speculum.conf, owned by root with permissions 0600:
machine https://mirror.example.com/debian/
login <username>
password <token>
machine https://mirror.example.com/debian-security/
login <username>
password <token>
Add matching path-specific entries for each protected Ubuntu mirror you use. The password is a package token, not the dashboard password. Anonymous mirrors need no credential entry. Both dashboards provide copyable configurations.
Replace matching entries in /etc/apk/repositories, using your installed branch
in place of <branch> (for example v3.23):
https://mirror.example.com/alpine/<branch>/main
https://mirror.example.com/alpine/<branch>/community
For protected mirrors, save root's .netrc with permissions 0600, or select a
separate file using NETRC=/path/to/file when invoking APK:
machine mirror.example.com
login <username>
password <token>
The .netrc machine is the hostname, without a scheme, port, or repository path.
APK 2 and APK 3 clients retain their distribution keys and signature checks.
APT and APK files are served byte-for-byte without rewriting signatures,
decompressing indexes, or importing signing keys into Speculum. Native clients
verify signatures and package checksums; Speculum checks complete transfers and
SHA-256/SHA-512 APT by-hash URLs. The cache does not claim server-side signature
verification. Repository indexes are streamed to disk, including files larger
than 64 MiB. Inventory entries use repository-relative paths and mark indexes
and signatures as metadata.
Mutable indexes default to 300 seconds of freshness, then use upstream
ETag/Last-Modified revalidation. Cache-Control: no-cache forces revalidation.
APT pool/ files, by-hash indexes, and versioned APK packages remain cached
until configured retention expires or an administrator clears them. Failed
refreshes retain the previous complete generation; connection failures and
upstream 5xx responses permit a stale index response. Upstream 404s remain 404s.
When a client forces revalidation of an immutable cached file, connection/5xx
failures also fall back to that complete file while it remains within retention.
Only previously requested files are available offline. Expired signed metadata can still be rejected by the client; signature and expiry checks remain enabled. No repository crawl, full synchronization, or snapshot is performed.
New mirror settings and the additive distro cache table are populated at startup without resetting existing users, tokens, settings, or cached artifacts. TLS continues to terminate at your existing reverse proxy. Keep a pre-upgrade volume backup for rollback: older binaries that recognize only PyPI/npm/Cargo cannot read the expanded mirror settings.
- Metadata requests never prefetch package archives. Clients resolve normal dependencies, and each requested wheel, sdist, tarball, or crate is cached independently. Python metadata sidecars are downloaded only when requested.
- Concurrent requests for an artifact share one upstream download. Downloads stream to disk, verify advertised SHA-256/SHA-512/SHA-1 checksums, and become visible only after completion. No full archive is buffered in memory.
- An empty artifact lifetime means forever. Otherwise expiry is the original completed download timestamp plus the current registry lifetime. Extending the setting extends every artifact still stored; switching to forever removes expiry. Reads do not reset the timestamp. Already deleted files are downloaded again when requested.
- Expired artifacts cannot be served even before background cleanup deletes them. Manual deletion waits for active readers/downloads and preserves traffic statistics. A later client request may populate the cache again.
- Metadata is fresh for five minutes by default. Conditional refreshes reuse upstream ETags/Last-Modified. Retained metadata and unexpired files remain available during upstream failures; uncached/expired files fail with 502. Upstream 404s are returned as 404s rather than silently using old metadata.
- Changing an upstream URL selects a different cache identity. Download URLs from the previous upstream return 410 and require refreshed client metadata. Old files remain visible in the dashboard until cleared or expired.
- Each registry has an independent optional proxy.
socks5h://resolves DNS at the proxy;socks5://resolves it locally. HTTP(S) proxies are also accepted. Proxy credentials go in the separate username/password fields. In Docker, usehost.docker.internalto reach a proxy running on the host, rather than127.0.0.1. Disabling the proxy uses direct connections, ignoring system proxy environment variables. - Upstream and redirect hosts must match the configured upstream host or an explicitly allowed artifact host. The default CDN hosts are already included. Client credentials, cookies, and admin sessions are never forwarded upstream.
Overview and package pages show persistent lifetime request/hit/miss/download/ failure and bandwidth totals. HEAD requests do not count as file downloads. PyPI/npm/Cargo index metadata requests do not count as file downloads; Python sidecars and distro repository file GETs do. Upstream traffic includes bytes received during failed downloads. Client traffic counts file bytes emitted by the response stream; it is not an acknowledgement of client receipt. Storage totals reflect files currently on disk, including sidecars, and can briefly include expired files while cleanup is running. There are no daily history charts.
Dashboard passwords use Argon2id. Tokens and session IDs are stored hashed. Only
the first 12 characters of a token (including spm_) are also stored in plaintext
for identification in the user's Tokens table. The full token is shown only when
it is created. Dashboard mutations require CSRF tokens. Temporary passwords must
be changed before accessing dashboard features. Both session lifetimes are configurable,
defaulting to 12 hours. Proxy secrets are encrypted with the persisted
data-volume key, masked in settings responses, and never logged. Back up that
key together with the database. Ten failed login attempts per dashboard role from one direct
peer IP block login for five minutes; forwarded client-IP headers are not trusted.
Dashboard session cookies are HttpOnly/SameSite and Secure when the public URL uses
HTTPS. All public URLs come from the explicit setting rather than forwarded headers.
Personal lifetime statistics count artifact requests made with valid tokens, including on public registries. Anonymous or invalid credentials on public registries contribute only to global totals. Per-user accounting remains stored internally; dashboard views and summary/package APIs show global usage and shared cache storage to every signed-in account. Cache deletion and token revocation preserve history. Upstream downloads and bytes belong to the requesting user who performs the fetch; waiting users are counted only for their own requests and delivered bytes. Deleting a user preserves global traffic totals.
Speculum does not run database migrations. Changes to existing table schemas require a fresh application database. Token usage is stored in a separate table created automatically at startup, so enabling this feature preserves existing users and tokens. Distro cache resources use the same additive-table approach; existing configuration gains the new mirror defaults without a reset.
Requires Rust 1.89+ (including a native C toolchain for SQLite/TLS dependencies),
Bun, and Node 20.19+ or 22.12+.
On Windows with the GNU toolchain, include your MinGW bin directory on PATH
(for example C:\msys64\mingw64\bin).
cd dashboard
bun install --frozen-lockfile
bun run build
cd ..
cargo build --locked
Optionally set SPECULUM_LISTEN_IP and SPECULUM_LISTEN_PORT, then run
cargo run --locked. First startup generates and prints the initial admin
password; change it at /admin. Mirror users sign in at /account.
The process uses a fixed data/ directory beneath its working directory; only
one instance may open that directory. The Docker working directory is /app.
For UI development, run the backend on 8080 and bun run dev in dashboard/.
Rebuild dashboard assets and Rust to update the embedded production dashboard.
cargo fmt --check
cargo clippy --locked --all-targets -- -D warnings
cargo test --locked
cd dashboard
bun run typecheck
bun run build
tests/native_clients.py generates valid wheel/tarball/crate fixtures, installs
them with real uv/npm/Bun/Cargo clients and isolated caches, then repeats the
installs with upstream unavailable. It checks dependencies, npm scopes, Bun
lockfile tarball paths, authenticated Cargo source replacement, credential
isolation, and absence of unrelated cached versions. Run it after cargo build,
using an existing Python interpreter:
uv run --no-project --python 3.12 tests/native_clients.py
python tests/distro_clients.py additionally uses disposable Docker clients
(Debian 12, Ubuntu 24.04, Alpine 3.22/APK 2, and Alpine 3.23/APK 3) and generated,
signed APT/APK repositories. Docker and public images/tool downloads are needed
for fixture setup; tested installs have no public repository fallback. The suite
checks signatures, tampered packages, clean-cache offline installs, and uncached
offline failures. Signed fixture generation is cached under ignored
tests/.runtime/distro-fixtures/, keyed by fixture scripts. --keep leaves a
populated preview running until interrupted.
Use --keep for a populated local dashboard preview with disposable fixture
credentials. Rust integration tests additionally cover controlled-clock expiry,
retention changes, ranges/HEAD, corruption, concurrency, eviction coordination,
settings encryption, restart persistence, users/CSRF, and SOCKS5 authentication/DNS.
/healthz checks database availability. The binary's healthcheck subcommand
provides the Docker health check. Structured operational logs go to stdout;
RUST_LOG controls verbosity, including when running through Compose. Set it in
your shell or in a .env file alongside compose.yaml (see .env.example):
RUST_LOG=speculum=debug,tower_http=infoThe default is speculum=info,tower_http=info. Upstream metadata fetches,
artifact downloads, and HEAD probes log at INFO; registry requests and cache
outcomes log at DEBUG. Upstream failures and stale metadata fallback log at
WARN, while internal database and storage failures log at ERROR. Logs omit
credentials, headers, request bodies, and URL query strings. Recreate the
container with docker compose up -d after changing RUST_LOG.
A cold download failure is returned clearly and leaves no reusable partial cache file.
For a consistent backup, stop the service, copy all of /app/data with
docker compose cp speculum:/app/data ./speculum-backup, then restart it. Restore
the directory into the mounted volume while the service is stopped, preserving
ownership for container UID/GID 10001:10001. Do not omit secret.key or SQLite
WAL files from a copy. Do not run docker compose down --volumes on data you want
to retain.
To recover an admin password, stop Speculum and pass the replacement password on
stdin to speculum admin reset-password. With Compose, pipe the password from a
shell variable populated with a hidden password prompt:
docker compose stop speculum
printf '%s' "$NEW_ADMIN_PASSWORD" | docker compose run --rm -T speculum admin reset-password
docker compose start speculumThe recovery command invalidates every admin session and requires a password change on the next login. It leaves mirror users, configuration, and cached packages intact. Dashboard password changes invalidate prior sessions and issue a fresh session for the caller.