All notable changes to @testingbot/cli are documented here. Releases are published to npm from GitHub releases.
-
testingbot maestro --max-failures <n>: TestingBot stops each run once n flows (shards, when sharded) have failed and does not run the rest (the run'sfailFastAPI parameter). Enforced server-side, so it works with--async. -
testingbot xcuitest:--only-testingand--skip-testingchoose which tests run (TestClassorTestClass/testMethod, comma-separated, repeatable).--test-env KEY=VALUE/--test-argpass environment variables and launch arguments to the test runner process,--app-env KEY=VALUE/--app-argto the app under test. Invalid identifiers or variable names fail before anything is uploaded. -
testingbot failed-tests <build>: prints the names of the failed tests of a Selenium/Appium build, one per line (--jsonfor id, session ID, browser and OS), so CI can run only those again. Builds are addressed by name or numeric ID.
-
testingbot rerun --id <projectId>: re-runs flows of an earlier Maestro project on the same devices, without uploading anything again. By default every failed flow is re-run.--run <runIds>limits it to some runs (devices), and--flow <names>picks flows by name or ID, even ones that passed. Each flow runs again as a new attempt inside its original run, so the dashboard and reports show the latest result. The live table, exit code,--reportand--download-artifactscover only the re-run runs.--asyncqueues the re-run and exits. Ctrl-C detaches without cancelling. -
Proxy support:
--proxy <url>on every command, orHTTPS_PROXY/HTTP_PROXY/ALL_PROXYwithNO_PROXY. API calls, uploads, the live-updates WebSocket, the tunnel download and the tunnel itself go through it (HTTPS via CONNECT). Before, behind a proxy the pre-flight connectivity check failed with "No internet connection", and axios sent HTTPS requests to the proxy without CONNECT. -
Job summary and annotations in CI: in GitHub Actions a results table is appended to
$GITHUB_STEP_SUMMARYand every failed flow or run gets an::errorannotation; in Azure Pipelines the table is attached to the build and failures are logged as errors. Formaestro,espresso,xcuitestandstatus.TESTINGBOT_CI_SUMMARY=0turns it off. -
testingbot maestro --cancel-previous: before starting, cancels your queued or running runs of the same repository and branch or pull request from an earlier commit (cancelPreviouson the run API). Runs on the same commit are left alone. -
testingbot maestrodetects the repository URL, branch, commit and pull request in GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI and sends them as run metadata (repoUrlis new), so the dashboard can link to the commit and PR on any Git host. On a GitHub pull request the PR's head commit is used, not the merge commit. Explicit flags win. -
testingbot espressoandtestingbot xcuitesttake the same CI/CD metadata asmaestro:--branch,--pr-url,-m/--metadataand the CI environment detection, through one shared code path. -
--retry <count>,--download-artifacts [all|failed]and--artifacts-output-dirfortestingbot espressoandtestingbot xcuitest. A retried run is started again on the uploaded app; the last attempt decides the exit code, the--reportand the artifacts. Artifacts (video, logs, screenshots) are zipped with a folder per run, likemaestro. -
--timeout <minutes>fortestingbot maestro,espressoandxcuitest: how long to wait for results, counted from when the tests start and shared by every--retryattempt. Default 60. -
--fail-fastfortestingbot maestro,espressoandxcuitest: at the first failure that will not be retried, the runs still queued or running are cancelled. The command waits for them to stop, then reports as usual and fails with exit code 2. -
--priority high|normal|lowfortestingbot maestro,espressoandxcuitest: where the runs go in your own queue while they wait for a free parallel slot. High starts before normal, normal before low, then oldest first. It does not take slots from other accounts or raise your parallel limit. Needs the matching API change; older API versions ignore it.
- When the wait for results times out (after
--timeout, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (outcome: failedwith anerrorin--json). Before, it exited with code 1 and left the runs going.testingbot status --waitstill only stops watching and leaves the runs alone.
testingbot whoami: shows the account, plan, minutes left and concurrency, and which credentials are in use (options,TB_KEY/TB_SECRETor~/.testingbot).testingbot logout: deletes~/.testingbot, and warns whenTB_KEY/TB_SECRETstill apply.testingbot devices: lists device names and OS versions per framework, or real devices with their availability with--real(needs credentials), along with the flags the run commands take to select them.--disable-animations/--no-disable-animationsfortestingbot maestro, sent asmaestroOptions.disableAnimationsonly when one of them is given, soplatform.ios/android.disableAnimationsinconfig.yaml(and the device default) still apply otherwise.--debugfortestingbot espresso, which logs API responses the way it already did forxcuitestandmaestro.
- Maestro
assertScreenshotbaselines are now uploaded when the path has no extension (assertScreenshot: screenshots/home, which Maestro saves as.png) and when it uses the{ path }form. Before, these runs failed with "Screenshot file not found". includedPathsglobs in Maestro'sconfig.yamlare now bundled, which covers baselines whose paths contain variables (screenshots/${DEVICE}/home.png) and can't be resolved before the run.--download-artifacts failedfor Maestro now selects failed runs by the same rule as the exit code, so cancelled runs get their artifacts too.testingbot espresso --report htmlis now rejected before any upload. The Espresso API only serves JUnit reports, and before this the command exited 0 without writing a file.testingbot loginnow checks a randomstatevalue on the loopback callback, so another local process can't inject credentials or abort the login. An existing~/.testingbotis now set to mode0600.
--secret-env KEY=VALUE(repeatable) fortestingbot maestro: passes a variable to the flows like-e, but the value is sent asmaestroOptions.secretEnv, stored encrypted by the API and shown as***in the run options, the dashboard and the reports. A key given to both-eand--secret-envis rejected before any request is made. Values passed with-eremain visible in the run options and reports, so use--secret-envfor credentials.
--dry-runand--debugnever print--secret-envvalues; they appear as***in the printed request payload and API responses.
- follow-redirects 1.16.1, handlebars 4.7.10 (dev).
- A Maestro flow that passed (
status: DONE,success: 1) no longer counts as failed just becauseerror_messagesis non-empty. Maestro's log4j sometimes writes "Unable to write to stream ... maestro.log" to stderr at shutdown, and that turned passing runs into exit code 2. Pass/fail now comes fromstatusandsuccess, and cancelled flows and runs still fail.
--check-name <name>names the GitHub pull request checkTestingBot / <name>, so several runs on one commit (an iOS and an Android job) post separate checks that branch protection can require individually. Sent ascheckNamein the run metadata.
--app-url <url>fortestingbot maestro: download the app under test from an http(s) URL instead of a local file. All formats are supported (.apk,.apks,.ipa,.zip,.tar.gz); the type is taken from the URL path, then theContent-Dispositionheader, thenContent-Type. Expired signed links (401/403), such as EAS Build URLs after about an hour, are reported as such..tar.gzas an app format, local or downloaded, for Expo / EAS iOS simulator builds. The archive is extracted with the systemtarand the.appbundle inside (root orPayload/) is uploaded.testingbot upload <url>accepts URLs as well as paths.- Inside an EAS Build job the run is tagged with the EAS build id, profile and platform, and
EAS_BUILD_GIT_COMMIT_HASHis used as the commit SHA unless--commit-shais given.
- Only one app source may be given per run: a file,
--app-urlor--app-binary-id. - Temporary downloads and extractions are removed after the upload, including when the upload fails.
- @humanfs/node 0.16.8.
--jsonprints one JSON document on stdout (logs move to stderr);--json-fileand--json-file-namewrite it to disk while keeping console output. Every flow attempt is listed withattempt,latest,passed, device, timings and errors.- Exit codes now distinguish test failures (2) from CLI or infrastructure errors (1) for
maestro,espressoandxcuitest. With--json-filea failed run exits 0 so the file is the contract. testingbot status --id <projectId> [--wait],testingbot artifacts --id <projectId>,testingbot listandtestingbot upload <app>for working with runs started earlier, typically with--async.--app-binary-id <projectId>reuses an app uploaded earlier without re-uploading it. Every run prints its project ID after the upload.--device-matrix "<device>[:<version>][:real]"runs every flow on each listed device in one command, each device with its own run, results and retries.--branch,--pr-urland repeatable-m/--metadata KEY=VALUErun metadata;--exclude-flowsto leave files, directories or globs out of the bundle;--report allure.- Alternative spellings for common flags are accepted as hidden aliases (
--app-file,--flows,--apiKey,--device-model,--device-os,--format,--output,--test-suite-name). - The CLI logs when the server registered a GitHub status check for the run.
- JSON output reports the device that actually ran, not a wildcard request.
status --waitdetaches on Ctrl-C without cancelling the runs.
- Maestro fails instead of uploading a bundle with no runnable flow.
- Repeatable options no longer leak values between invocations.
- js-yaml, socket.io-parser and brace-expansion updates; workflow permissions for code scanning.
- Improved Maestro cancel signal handling.
- Maestro output distinguishes top-level flows from bundled subflows and warns when a flow runs both as top-level and as a subflow.
- axios 1.18.0, js-yaml 4.3.0 and transitive updates for Dependabot alerts.