You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d9c0539
Browse filesBrowse the repository at this point in the historyBrowse files
Add --timeout: cancel unfinished runs and fail when the wait for results runs out
--timeout <minutes> for maestro, espresso and xcuitest sets one deadline
for the wait, shared by every --retry attempt (default 60 minutes, as
before). At the deadline the CLI cancels the runs still queued or running
and returns a failed result with the results so far and an error, instead
of throwing and leaving the runs going. status --wait only stops watching.
Copy file name to clipboardExpand all lines: CHANGELOG.md
+5Lines changed: 5 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,11 @@ All notable changes to `@testingbot/cli` are documented here. Releases are publi
11
11
-`testingbot maestro` detects the repository URL, branch, commit and pull request in GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI and sends them as run metadata (`repoUrl` is new), so the dashboard can link to the commit and PR on any Git host. On a GitHub pull request the PR's head commit is used, not the merge commit. Explicit flags win.
12
12
-`testingbot espresso` and `testingbot xcuitest` take the same CI/CD metadata as `maestro`: `--branch`, `--pr-url`, `-m/--metadata` and the CI environment detection, through one shared code path.
13
13
-`--retry <count>`, `--download-artifacts [all|failed]` and `--artifacts-output-dir` for `testingbot espresso` and `testingbot xcuitest`. A retried run is started again on the uploaded app; the last attempt decides the exit code, the `--report` and the artifacts. Artifacts (video, logs, screenshots) are zipped with a folder per run, like `maestro`.
14
+
-`--timeout <minutes>` for `testingbot maestro`, `espresso` and `xcuitest`: how long to wait for results, counted from when the tests start and shared by every `--retry` attempt. Default 60.
15
+
16
+
### Changed
17
+
18
+
- When the wait for results times out (after `--timeout`, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (`outcome: failed` with an `error` in `--json`). Before, it exited with code 1 and left the runs going. `testingbot status --wait` still only stops watching and leaves the runs alone.
|`--shard-split <number>`| Split flows into N parallel sessions for faster execution |
189
-
|`--retry <count>`| Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. |
190
-
|`--ignore-checksum-check`| Skip checksum verification and always upload the app |
|`--shard-split <number>`| Split flows into N parallel sessions for faster execution |
189
+
|`--retry <count>`| Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. |
190
+
|`--timeout <minutes>`| Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
191
+
|`--ignore-checksum-check`| Skip checksum verification and always upload the app |
191
192
192
193
> **Note on `--retry`:** a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a `↻` icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the **last** attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via `--report`.
| `--async` | Start tests and exit without waiting for results |
528
-
| `-q, --quiet` | Suppress progress output |
529
-
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
530
-
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json`in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
531
-
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
532
-
| `--report <format>` | Download report after completion: junit (Espresso has no HTML report) |
533
-
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
534
-
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
535
-
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all`(default) or `failed` |
536
-
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
| `--async` | Start tests and exit without waiting for results |
529
+
| `-q, --quiet` | Suppress progress output |
530
+
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
531
+
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json`in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
532
+
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
533
+
| `--report <format>` | Download report after completion: junit (Espresso has no HTML report) |
534
+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
535
+
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
536
+
| `--timeout <minutes>` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
537
+
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all`(default) or `failed` |
538
+
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
| `--async` | Start tests and exit without waiting for results |
635
-
| `-q, --quiet` | Suppress progress output |
636
-
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
637
-
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json`in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
638
-
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
639
-
| `--report <format>` | Download report after completion: html or junit |
640
-
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
641
-
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
642
-
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all`(default) or `failed` |
643
-
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
| `--async` | Start tests and exit without waiting for results |
637
+
| `-q, --quiet` | Suppress progress output |
638
+
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
639
+
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json`in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
640
+
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
641
+
| `--report <format>` | Download report after completion: html or junit |
642
+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
643
+
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
644
+
| `--timeout <minutes>` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
645
+
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all`(default) or `failed` |
646
+
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
0 commit comments