Skip to content

Commit d9c0539

Browse files
Add --timeout: cancel unfinished runs and fail when the wait for results runs out
--timeout <minutes> for maestro, espresso and xcuitest sets one deadline for the wait, shared by every --retry attempt (default 60 minutes, as before). At the deadline the CLI cancels the runs still queued or running and returns a failed result with the results so far and an error, instead of throwing and leaving the runs going. status --wait only stops watching.
1 parent 3373aa4 commit d9c0539

14 files changed

Lines changed: 345 additions & 50 deletions

‎CHANGELOG.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@ All notable changes to `@testingbot/cli` are documented here. Releases are publi
1111
- `testingbot maestro` detects the repository URL, branch, commit and pull request in GitHub Actions, GitLab CI, Bitbucket Pipelines, Azure Pipelines and CircleCI and sends them as run metadata (`repoUrl` is new), so the dashboard can link to the commit and PR on any Git host. On a GitHub pull request the PR's head commit is used, not the merge commit. Explicit flags win.
1212
- `testingbot espresso` and `testingbot xcuitest` take the same CI/CD metadata as `maestro`: `--branch`, `--pr-url`, `-m/--metadata` and the CI environment detection, through one shared code path.
1313
- `--retry <count>`, `--download-artifacts [all|failed]` and `--artifacts-output-dir` for `testingbot espresso` and `testingbot xcuitest`. A retried run is started again on the uploaded app; the last attempt decides the exit code, the `--report` and the artifacts. Artifacts (video, logs, screenshots) are zipped with a folder per run, like `maestro`.
14+
- `--timeout <minutes>` for `testingbot maestro`, `espresso` and `xcuitest`: how long to wait for results, counted from when the tests start and shared by every `--retry` attempt. Default 60.
15+
16+
### Changed
17+
18+
- When the wait for results times out (after `--timeout`, or the default 60 minutes), the CLI now cancels the runs that are still queued or running, prints the results so far and exits with code 2 (`outcome: failed` with an `error` in `--json`). Before, it exited with code 1 and left the runs going. `testingbot status --wait` still only stops watching and leaves the runs alone.
1419

1520
## 1.4.3 - 2026-10-27
1621

‎README.md‎

Lines changed: 32 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -183,11 +183,12 @@ Values passed with `-e` are stored in plain text: they are returned in the run's
183183

184184
**Advanced Options:**
185185

186-
| Option | Description |
187-
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
188-
| `--shard-split <number>` | Split flows into N parallel sessions for faster execution |
189-
| `--retry <count>` | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. |
190-
| `--ignore-checksum-check` | Skip checksum verification and always upload the app |
186+
| Option | Description |
187+
| ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
188+
| `--shard-split <number>` | Split flows into N parallel sessions for faster execution |
189+
| `--retry <count>` | Retry failed flows up to N times (0-2, default 0). Re-runs only the flows (or shards) that failed, the moment they fail, while the rest of the run continues. Cannot be combined with `--async`. |
190+
| `--timeout <minutes>` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
191+
| `--ignore-checksum-check` | Skip checksum verification and always upload the app |
191192

192193
> **Note on `--retry`:** a failed flow/shard is retried immediately — as soon as it fails — without waiting for the other flows in the run to finish. Retry attempts appear live in the flow table marked with a `↻` icon. Each flow is retried independently up to N times, stopping as soon as that flow passes. Pass/fail uses the result of the **last** attempt per flow (last-attempt-wins), consistently across the CLI exit code, the TestingBot dashboard, and reports downloaded via `--report`.
193194
@@ -522,18 +523,19 @@ testingbot espresso [appFile] [testAppFile] [options]
522523

523524
**Output Options:**
524525

525-
| Option | Description |
526-
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
527-
| `--async` | Start tests and exit without waiting for results |
528-
| `-q, --quiet` | Suppress progress output |
529-
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
530-
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
531-
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
532-
| `--report <format>` | Download report after completion: junit (Espresso has no HTML report) |
533-
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
534-
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
535-
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` |
536-
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
526+
| Option | Description |
527+
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
528+
| `--async` | Start tests and exit without waiting for results |
529+
| `-q, --quiet` | Suppress progress output |
530+
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
531+
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
532+
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
533+
| `--report <format>` | Download report after completion: junit (Espresso has no HTML report) |
534+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
535+
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
536+
| `--timeout <minutes>` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
537+
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` |
538+
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
537539

538540
**Examples:**
539541

@@ -629,18 +631,19 @@ testingbot xcuitest [appFile] [testAppFile] [options]
629631

630632
**Output Options:**
631633

632-
| Option | Description |
633-
| ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
634-
| `--async` | Start tests and exit without waiting for results |
635-
| `-q, --quiet` | Suppress progress output |
636-
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
637-
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
638-
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
639-
| `--report <format>` | Download report after completion: html or junit |
640-
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
641-
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
642-
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` |
643-
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
634+
| Option | Description |
635+
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
636+
| `--async` | Start tests and exit without waiting for results |
637+
| `-q, --quiet` | Suppress progress output |
638+
| `--json` | Print results as a single JSON document on stdout (logs move to stderr). Implies `--quiet`. Exit code 2 when tests fail |
639+
| `--json-file` | Write results as JSON to a file (default: `<appId>_testingbot.json` in the current directory). Implies `--quiet`. Exit code stays 0 when tests fail so the pipeline can gate on the file |
640+
| `--json-file-name <path>` | Custom path for the JSON results file (requires `--json-file`) |
641+
| `--report <format>` | Download report after completion: html or junit |
642+
| `--report-output-dir <path>` | Directory to save reports (required with --report) |
643+
| `--retry <count>` | Run a failed test again up to N times (0-2). The last attempt decides the result, the report and the artifacts |
644+
| `--timeout <minutes>` | Stop waiting for results after N minutes (default 60), counted from when the tests start and including `--retry` attempts. Unfinished runs are stopped and the command fails with exit code 2. Cannot be combined with `--async`. |
645+
| `--download-artifacts [mode]` | Download video, logs and screenshots as a zip after completion: `all` (default) or `failed` |
646+
| `--artifacts-output-dir <path>` | Directory for the artifacts zip (defaults to the current directory) |
644647

645648
**Examples:**
646649

‎src/cli.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -524,6 +524,11 @@ program
524524
(val) => parseInt(val, 10),
525525
0,
526526
)
527+
.option(
528+
'--timeout <minutes>',
529+
'Stop waiting after N minutes (default 60), including retries. Unfinished runs are stopped and the command fails.',
530+
(val) => Number(val),
531+
)
527532
.option(
528533
'--download-artifacts [mode]',
529534
'Download the video, logs and screenshots after completion. Mode: all (default) or failed.',
@@ -632,6 +637,7 @@ program
632637
report: args.report,
633638
reportOutputDir: args.reportOutputDir,
634639
retry: args.retry,
640+
timeout: args.timeout,
635641
downloadArtifacts:
636642
args.downloadArtifacts === true
637643
? 'all'
@@ -801,6 +807,11 @@ program
801807
(val) => parseInt(val, 10),
802808
0,
803809
)
810+
.option(
811+
'--timeout <minutes>',
812+
'Stop waiting after N minutes (default 60), including retries. Unfinished runs are stopped and the command fails.',
813+
(val) => Number(val),
814+
)
804815
.option(
805816
'--async',
806817
'Start tests and exit immediately without waiting for results.',
@@ -1040,6 +1051,7 @@ program
10401051
ignoreChecksumCheck: args.ignoreChecksumCheck,
10411052
shardSplit: args.shardSplit,
10421053
retry: args.retry,
1054+
timeout: args.timeout,
10431055
cancelPrevious: args.cancelPrevious,
10441056
debug: args.debug,
10451057
googlePlayStore: args.googlePlay,
@@ -1141,6 +1153,11 @@ program
11411153
(val) => parseInt(val, 10),
11421154
0,
11431155
)
1156+
.option(
1157+
'--timeout <minutes>',
1158+
'Stop waiting after N minutes (default 60), including retries. Unfinished runs are stopped and the command fails.',
1159+
(val) => Number(val),
1160+
)
11441161
.option(
11451162
'--download-artifacts [mode]',
11461163
'Download the video, logs and screenshots after completion. Mode: all (default) or failed.',
@@ -1242,6 +1259,7 @@ program
12421259
report: args.report,
12431260
reportOutputDir: args.reportOutputDir,
12441261
retry: args.retry,
1262+
timeout: args.timeout,
12451263
downloadArtifacts:
12461264
args.downloadArtifacts === true
12471265
? 'all'

0 commit comments

Comments
 (0)