Skip to content

Releases

1.5.0

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 05 Nov 10:20

Added

  • testingbot rerun --id <projectId>: re-runs flows of an earlier Maestro project on the same devices, without uploading anything again. By default every failed flow is re-run. --run <runIds> limits it to some runs (devices), and --flow <names> picks flows by name or ID, even ones that passed. Each flow runs again as a new attempt inside its original run, so the dashboard and reports show the latest result. The live table, exit code, --report and --download-artifacts cover only the re-run runs. --async queues the re-run and exits. Ctrl-C detaches without cancelling.

  • Proxy support: --proxy <url> on every command, or HTTPS_PROXY / HTTP_PROXY / ALL_PROXY with NO_PROXY. API calls, uploads, the live-updates WebSocket, the tunnel download and the tunnel itself go through it (HTTPS via CONNECT). Before, behind a proxy the pre-flight connectivity check failed with "No internet connection", and axios sent HTTPS requests to the proxy without CONNECT.

  • Job summary and annotations in CI: in GitHub Actions a results table is appended to $GITHUB_STEP_SUMMARY and every failed flow or run gets an ::error annotation; in Azure Pipelines the table is attached to the build and failures are logged as errors. For maestro, espresso, xcuitest and status. TESTINGBOT_CI_SUMMARY=0 turns it off.

  • testingbot maestro --cancel-previous: before starting, cancels your queued or running runs of the same repository and branch or pull request from an earlier commit (cancelPrevious on the run API). Runs on the same commit are left alone.

  • testingbot maestro detects the repository URL, branch, commit and pull request in GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI and sends them as run metadata (repoUrl is new), so the dashboard can link to the commit and PR on any Git host. On a GitHub pull request the PR's head commit is used, not the merge commit. Explicit flags win.

  • testingbot espresso and testingbot xcuitest take the same CI/CD metadata as maestro: --branch, --pr-url, -m/--metadata and the CI environment detection, through one shared code path.

  • --retry <count>, --download-artifacts [all|failed] and --artifacts-output-dir for testingbot espresso and testingbot xcuitest. A retried run is started again on the uploaded app; the last attempt decides the exit code, the --report and the artifacts. Artifacts (video, logs, screenshots) are zipped with a folder per run, like maestro.

  • --timeout <minutes> for testingbot maestro, espresso and xcuitest: how long to wait for results, counted from when the tests start and shared by every --retry attempt. Default 60.

  • --fail-fast for testingbot maestro, espresso and xcuitest: at the first failure that will not be retried, the runs still queued or running are cancelled. The command waits for them to stop, then reports as usual and fails with exit code 2.

  • --priority high|normal|low for testingbot maestro, espresso and xcuitest: where the runs go in your own queue while they wait for a free parallel slot. High starts before normal, normal before low, then oldest first. It does not take slots from other accounts or raise your parallel limit. Needs the matching API change; older API versions ignore it.

Changed

  • When the wait for results times out (after --timeout, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (outcome: failed with an error in --json). Before, it exited with code 1 and left the runs going. testingbot status --wait still only stops watching and leaves the runs alone.

1.4.3

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 27 Oct 12:57

Added

  • testingbot whoami: shows the account, plan, minutes left and concurrency, and which credentials are in use (options, TB_KEY/TB_SECRET or ~/.testingbot).
  • testingbot logout: deletes ~/.testingbot, and warns when TB_KEY/TB_SECRET still apply.
  • testingbot devices: lists device names and OS versions per framework, or real devices with their availability with --real (needs credentials), along with the flags the run commands take to select them.
  • --disable-animations / --no-disable-animations for testingbot maestro, sent as maestroOptions.disableAnimations only when one of them is given, so platform.ios/android.disableAnimations in config.yaml (and the device default) still apply otherwise.
  • --debug for testingbot espresso, which logs API responses the way it already did for xcuitest and maestro.

Fixed

  • Maestro assertScreenshot baselines are now uploaded when the path has no extension (assertScreenshot: screenshots/home, which Maestro saves as .png) and when it uses the { path } form. Before, these runs failed with "Screenshot file not found".
  • includedPaths globs in Maestro's config.yaml are now bundled, which covers baselines whose paths contain variables (screenshots/${DEVICE}/home.png) and can't be resolved before the run.
  • --download-artifacts failed for Maestro now selects failed runs by the same rule as the exit code, so cancelled runs get their artifacts too.
  • testingbot espresso --report html is now rejected before any upload. The Espresso API only serves JUnit reports, and before this the command exited 0 without writing a file.
  • testingbot login now checks a random state value on the loopback callback, so another local process can't inject credentials or abort the login. An existing ~/.testingbot is now set to mode 0600.

Full Changelog: 1.4.2...1.4.3

1.4.2

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 25 Oct 13:18

Added

  • --secret-env KEY=VALUE (repeatable) for testingbot maestro: passes a variable to the flows like -e, but the value is sent as maestroOptions.secretEnv, stored encrypted by the API and shown as *** in the run options, the dashboard and the reports. A key given to both -e and --secret-env is rejected before any request is made. Values passed with -e remain visible in the run options and reports, so use --secret-env for credentials.

Changed

  • --dry-run and --debug never print --secret-env values; they appear as *** in the printed request payload and API responses.

Dependencies

  • follow-redirects 1.16.0 → 1.16.1
  • handlebars 4.7.9 → 4.7.10 (dev)

Full Changelog: 1.4.1...1.4.2

1.4.1

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 01 Oct 11:26
ee59411

Fixed

  • A Maestro flow that passed (status: DONE, success: 1) no longer counts as failed just because error_messages is non-empty. Maestro's log4j sometimes writes "Unable to write to stream ... maestro.log" to stderr at shutdown, and that turned passing runs into exit code 2. Pass/fail now comes from status and success, and cancelled flows and runs still fail.

Dependencies

  • axios 1.18.0 → 1.20.0
  • brace-expansion bump

Full Changelog: 1.4.0...1.4.1

1.4.0

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 07 Sep 11:17
3bd2255

Added

  • --check-name <name> for testingbot maestro: names the GitHub pull request check TestingBot / <name> (for example iOS, Android) so several runs on one commit, such as one job per platform, post separate checks that can each be required under branch protection. Runs without it keep TestingBot / tests. Needs the matching TestingBot API deployment; the GitHub Action (check-name) and the EAS Workflows wrapper (TB_CHECK_NAME) expose the same option.

Full Changelog: 1.3.0...1.4.0

1.3.0

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 04 Sep 12:53
110c1b8

Highlights

Expo and EAS Build artifacts are now a first-class input, and apps can be pulled from any download URL.

Added

  • --app-url <url> downloads the app under test instead of reading a local file. All formats: .apk, .apks, .ipa, .zip, .tar.gz. The type is taken from the URL path, then Content-Disposition, then Content-Type. Expired signed links (401/403), such as EAS Build URLs after about an hour, are reported as such. Every positional argument is then a flow.
  • .tar.gz apps, local or downloaded, for EAS iOS simulator builds: extracted with the system tar, the .app bundle inside is uploaded.
  • testingbot upload <url> accepts URLs.
  • EAS Build awareness: inside an EAS job the run is tagged with the build id, profile and platform, and EAS_BUILD_GIT_COMMIT_HASH is the default commit SHA.
  • CHANGELOG.md in the repository.

Changed

  • Only one app source may be given per run: a file, --app-url or --app-binary-id.
  • Temporary downloads and extractions are removed after the upload, including on failure.

Dependencies

  • @humanfs/node 0.16.8.

Full Changelog: 1.2.0...1.3.0

1.2.0

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 02 Sep 12:45
3c322a6

Highlights

This release makes testingbot maestro a first-class citizen in CI pipelines: machine-readable results, meaningful exit codes, commands to come back for async runs, app reuse, device matrices, and the plumbing the new TestingBot Maestro GitHub Action builds on.

Machine-readable output and exit codes

  • --json prints one JSON document on stdout (logs move to stderr); --json-file / --json-file-name write it to disk while keeping console output. Every flow attempt is listed with attempt, latest, passed, device, timings and errors.
  • Exit codes now distinguish test failures (2) from CLI or infrastructure errors (1) for maestro, espresso and xcuitest. With --json-file a failed run still exits 0 so the file is the contract.
  • JSON reports the device that actually ran, not a wildcard request. (#31, #36)

New commands for async runs

  • testingbot status --id <projectId> [--wait]: snapshot of every run and flow; --wait attaches with the live flow table. Ctrl-C detaches without cancelling.
  • testingbot artifacts --id <projectId>: download --report and/or --download-artifacts for a finished project.
  • testingbot list [--count] [--offset]: recent projects, newest first.
  • testingbot upload <app>: upload an app once and get a project ID. (#32, #33)

Reuse an uploaded app

  • --app-binary-id <projectId> runs new flows against an app uploaded earlier without re-uploading it; platform is taken from the stored app. Every run prints its project ID after the upload. Requires the matching TestingBot API deployment. (#33)

Device matrix

  • --device-matrix "<device>[:<version>][:real]" (comma-separated or repeatable) runs every flow on each listed device in one command, each device with its own run, results and retries. The CLI prints devices × flows before submitting. (#35)

Drop-in flags and reports

  • --branch, --pr-url and repeatable -m/--metadata KEY=VALUE run metadata.
  • --exclude-flows to leave files, directories or globs out of the bundle.
  • --report allure writes Allure result files next to the existing junit/html formats.
  • Alternative spellings for common flags are accepted as hidden aliases (--app-file, --flows, --apiKey, --device-model, --device-os, --format, --output, --test-suite-name), so command lines written for other Maestro runners work unchanged. (#34)

GitHub PR checks

  • When a run carries repository and commit metadata and the TestingBot GitHub App is installed, the server posts a TestingBot / tests check on the pull request; the CLI logs when that happened. (#37)

Fixes and maintenance

  • Maestro: fail instead of uploading a bundle with no runnable flow.
  • Repeatable options no longer leak values between invocations.
  • Tests: the --exclude-flows directory test is path-separator agnostic (fixes Windows CI).
  • Dependency and code-scanning updates (js-yaml, socket.io-parser, brace-expansion; workflow permissions).

Full Changelog: 1.1.1...1.2.0

1.1.1

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 27 Jul 13:24
  • Improve Maestro cancel signal handling
  • Maestro: distinguish top-level flows from bundled subflows in the output
  • Maestro: warn when a flow runs twice as both top-level and subflow
  • Security: update transitive dependencies (brace-expansion, js-yaml, @babel/core) to resolve Dependabot alerts

Full Changelog: 1.1.0...1.1.1

1.1.0

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 03 Jun 12:30

Maestro

  • Add --other-app argument to install other apps on the remote TestingBot devices
  • Add --retry {x<2} argument to automatically retry failing Maestro flows
  • Change structure of artifacts zip

CLI

  • Improve --debug

XCUITest + Espresso

  • Report improvements

What's Changed

Full Changelog: 1.0.9...1.1.0

1.0.9

Choose a tag to compare

@jochen-testingbot jochen-testingbot released this 29 Apr 10:58

Maestro

  • Improve --download-artifacts
  • Improve Maestro UI flow table updating
  • Add --groups CLI allowing you to add 'group tags' which are visible in the TestingBot Dashboard
  • Add --google-play CLI option, to indicate you want to run on Google Play Android Emulators

Full Changelog: 1.0.8...1.0.9