Repository navigation
Releases
1.5.0
Added
-
testingbot rerun --id <projectId>: re-runs flows of an earlier Maestro project on the same devices, without uploading anything again. By default every failed flow is re-run.--run <runIds>limits it to some runs (devices), and--flow <names>picks flows by name or ID, even ones that passed. Each flow runs again as a new attempt inside its original run, so the dashboard and reports show the latest result. The live table, exit code,--reportand--download-artifactscover only the re-run runs.--asyncqueues the re-run and exits. Ctrl-C detaches without cancelling. -
Proxy support:
--proxy <url>on every command, orHTTPS_PROXY/HTTP_PROXY/ALL_PROXYwithNO_PROXY. API calls, uploads, the live-updates WebSocket, the tunnel download and the tunnel itself go through it (HTTPS via CONNECT). Before, behind a proxy the pre-flight connectivity check failed with "No internet connection", and axios sent HTTPS requests to the proxy without CONNECT. -
Job summary and annotations in CI: in GitHub Actions a results table is appended to
$GITHUB_STEP_SUMMARYand every failed flow or run gets an::errorannotation; in Azure Pipelines the table is attached to the build and failures are logged as errors. Formaestro,espresso,xcuitestandstatus.TESTINGBOT_CI_SUMMARY=0turns it off. -
testingbot maestro --cancel-previous: before starting, cancels your queued or running runs of the same repository and branch or pull request from an earlier commit (cancelPreviouson the run API). Runs on the same commit are left alone. -
testingbot maestrodetects the repository URL, branch, commit and pull request in GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI and sends them as run metadata (repoUrlis new), so the dashboard can link to the commit and PR on any Git host. On a GitHub pull request the PR's head commit is used, not the merge commit. Explicit flags win. -
testingbot espressoandtestingbot xcuitesttake the same CI/CD metadata asmaestro:--branch,--pr-url,-m/--metadataand the CI environment detection, through one shared code path. -
--retry <count>,--download-artifacts [all|failed]and--artifacts-output-dirfortestingbot espressoandtestingbot xcuitest. A retried run is started again on the uploaded app; the last attempt decides the exit code, the--reportand the artifacts. Artifacts (video, logs, screenshots) are zipped with a folder per run, likemaestro. -
--timeout <minutes>fortestingbot maestro,espressoandxcuitest: how long to wait for results, counted from when the tests start and shared by every--retryattempt. Default 60. -
--fail-fastfortestingbot maestro,espressoandxcuitest: at the first failure that will not be retried, the runs still queued or running are cancelled. The command waits for them to stop, then reports as usual and fails with exit code 2. -
--priority high|normal|lowfortestingbot maestro,espressoandxcuitest: where the runs go in your own queue while they wait for a free parallel slot. High starts before normal, normal before low, then oldest first. It does not take slots from other accounts or raise your parallel limit. Needs the matching API change; older API versions ignore it.
Changed
- When the wait for results times out (after
--timeout, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (outcome: failedwith anerrorin--json). Before, it exited with code 1 and left the runs going.testingbot status --waitstill only stops watching and leaves the runs alone.
1.4.3
Added
testingbot whoami: shows the account, plan, minutes left and concurrency, and which credentials are in use (options,TB_KEY/TB_SECRETor~/.testingbot).testingbot logout: deletes~/.testingbot, and warns whenTB_KEY/TB_SECRETstill apply.testingbot devices: lists device names and OS versions per framework, or real devices with their availability with--real(needs credentials), along with the flags the run commands take to select them.--disable-animations/--no-disable-animationsfortestingbot maestro, sent asmaestroOptions.disableAnimationsonly when one of them is given, soplatform.ios/android.disableAnimationsinconfig.yaml(and the device default) still apply otherwise.--debugfortestingbot espresso, which logs API responses the way it already did forxcuitestandmaestro.
Fixed
- Maestro
assertScreenshotbaselines are now uploaded when the path has no extension (assertScreenshot: screenshots/home, which Maestro saves as.png) and when it uses the{ path }form. Before, these runs failed with "Screenshot file not found". includedPathsglobs in Maestro'sconfig.yamlare now bundled, which covers baselines whose paths contain variables (screenshots/${DEVICE}/home.png) and can't be resolved before the run.--download-artifacts failedfor Maestro now selects failed runs by the same rule as the exit code, so cancelled runs get their artifacts too.testingbot espresso --report htmlis now rejected before any upload. The Espresso API only serves JUnit reports, and before this the command exited 0 without writing a file.testingbot loginnow checks a randomstatevalue on the loopback callback, so another local process can't inject credentials or abort the login. An existing~/.testingbotis now set to mode0600.
Full Changelog: 1.4.2...1.4.3
1.4.2
Added
--secret-env KEY=VALUE(repeatable) fortestingbot maestro: passes a variable to the flows like-e, but the value is sent asmaestroOptions.secretEnv, stored encrypted by the API and shown as***in the run options, the dashboard and the reports. A key given to both-eand--secret-envis rejected before any request is made. Values passed with-eremain visible in the run options and reports, so use--secret-envfor credentials.
Changed
--dry-runand--debugnever print--secret-envvalues; they appear as***in the printed request payload and API responses.
Dependencies
- follow-redirects 1.16.0 → 1.16.1
- handlebars 4.7.9 → 4.7.10 (dev)
Full Changelog: 1.4.1...1.4.2
1.4.1
Fixed
- A Maestro flow that passed (
status: DONE,success: 1) no longer counts as failed just becauseerror_messagesis non-empty. Maestro's log4j sometimes writes "Unable to write to stream ... maestro.log" to stderr at shutdown, and that turned passing runs into exit code 2. Pass/fail now comes fromstatusandsuccess, and cancelled flows and runs still fail.
Dependencies
- axios 1.18.0 → 1.20.0
- brace-expansion bump
Full Changelog: 1.4.0...1.4.1
1.4.0
Added
--check-name <name>fortestingbot maestro: names the GitHub pull request checkTestingBot / <name>(for exampleiOS,Android) so several runs on one commit, such as one job per platform, post separate checks that can each be required under branch protection. Runs without it keepTestingBot / tests. Needs the matching TestingBot API deployment; the GitHub Action (check-name) and the EAS Workflows wrapper (TB_CHECK_NAME) expose the same option.
Full Changelog: 1.3.0...1.4.0
1.3.0
Highlights
Expo and EAS Build artifacts are now a first-class input, and apps can be pulled from any download URL.
Added
--app-url <url>downloads the app under test instead of reading a local file. All formats:.apk,.apks,.ipa,.zip,.tar.gz. The type is taken from the URL path, thenContent-Disposition, thenContent-Type. Expired signed links (401/403), such as EAS Build URLs after about an hour, are reported as such. Every positional argument is then a flow..tar.gzapps, local or downloaded, for EAS iOS simulator builds: extracted with the systemtar, the.appbundle inside is uploaded.testingbot upload <url>accepts URLs.- EAS Build awareness: inside an EAS job the run is tagged with the build id, profile and platform, and
EAS_BUILD_GIT_COMMIT_HASHis the default commit SHA. CHANGELOG.mdin the repository.
Changed
- Only one app source may be given per run: a file,
--app-urlor--app-binary-id. - Temporary downloads and extractions are removed after the upload, including on failure.
Dependencies
- @humanfs/node 0.16.8.
Full Changelog: 1.2.0...1.3.0
1.2.0
Highlights
This release makes testingbot maestro a first-class citizen in CI pipelines: machine-readable results, meaningful exit codes, commands to come back for async runs, app reuse, device matrices, and the plumbing the new TestingBot Maestro GitHub Action builds on.
Machine-readable output and exit codes
--jsonprints one JSON document on stdout (logs move to stderr);--json-file/--json-file-namewrite it to disk while keeping console output. Every flow attempt is listed withattempt,latest,passed, device, timings and errors.- Exit codes now distinguish test failures (2) from CLI or infrastructure errors (1) for
maestro,espressoandxcuitest. With--json-filea failed run still exits 0 so the file is the contract. - JSON reports the device that actually ran, not a wildcard request. (#31, #36)
New commands for async runs
testingbot status --id <projectId> [--wait]: snapshot of every run and flow;--waitattaches with the live flow table. Ctrl-C detaches without cancelling.testingbot artifacts --id <projectId>: download--reportand/or--download-artifactsfor a finished project.testingbot list [--count] [--offset]: recent projects, newest first.testingbot upload <app>: upload an app once and get a project ID. (#32, #33)
Reuse an uploaded app
--app-binary-id <projectId>runs new flows against an app uploaded earlier without re-uploading it; platform is taken from the stored app. Every run prints its project ID after the upload. Requires the matching TestingBot API deployment. (#33)
Device matrix
--device-matrix "<device>[:<version>][:real]"(comma-separated or repeatable) runs every flow on each listed device in one command, each device with its own run, results and retries. The CLI prints devices × flows before submitting. (#35)
Drop-in flags and reports
--branch,--pr-urland repeatable-m/--metadata KEY=VALUErun metadata.--exclude-flowsto leave files, directories or globs out of the bundle.--report allurewrites Allure result files next to the existing junit/html formats.- Alternative spellings for common flags are accepted as hidden aliases (
--app-file,--flows,--apiKey,--device-model,--device-os,--format,--output,--test-suite-name), so command lines written for other Maestro runners work unchanged. (#34)
GitHub PR checks
- When a run carries repository and commit metadata and the TestingBot GitHub App is installed, the server posts a
TestingBot / testscheck on the pull request; the CLI logs when that happened. (#37)
Fixes and maintenance
- Maestro: fail instead of uploading a bundle with no runnable flow.
- Repeatable options no longer leak values between invocations.
- Tests: the
--exclude-flowsdirectory test is path-separator agnostic (fixes Windows CI). - Dependency and code-scanning updates (js-yaml, socket.io-parser, brace-expansion; workflow permissions).
Full Changelog: 1.1.1...1.2.0
1.1.1
- Improve Maestro cancel signal handling
- Maestro: distinguish top-level flows from bundled subflows in the output
- Maestro: warn when a flow runs twice as both top-level and subflow
- Security: update transitive dependencies (brace-expansion, js-yaml, @babel/core) to resolve Dependabot alerts
- Bump axios from 1.16.0 to 1.18.0 by @dependabot[bot] in #23
- Bump js-yaml from 4.2.0 to 4.3.0 by @dependabot[bot] in #24
Full Changelog: 1.1.0...1.1.1
1.1.0
Maestro
- Add
--other-appargument to install other apps on the remote TestingBot devices - Add
--retry {x<2}argument to automatically retry failing Maestro flows - Change structure of artifacts zip
CLI
- Improve
--debug
XCUITest + Espresso
- Report improvements
What's Changed
- Bump axios from 1.15.0 to 1.15.2 by @dependabot[bot] in #17
- Bump axios from 1.15.2 to 1.16.0 by @dependabot[bot] in #18
- Bump ws and engine.io-client by @dependabot[bot] in #19
Full Changelog: 1.0.9...1.1.0
1.0.9
Maestro
- Improve
--download-artifacts - Improve Maestro UI flow table updating
- Add
--groupsCLI allowing you to add 'group tags' which are visible in the TestingBot Dashboard - Add
--google-playCLI option, to indicate you want to run on Google Play Android Emulators
Full Changelog: 1.0.8...1.0.9