Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions .agents/skills/sim-use/SKILL.md

Large diffs are not rendered by default.

104 changes: 104 additions & 0 deletions .agents/skills/sim-use/references/batch-reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# sim-use Batch Reference

Use this reference when generating or reviewing `sim-use ios batch` commands.

## Supported step commands
- `tap`
- `swipe` (`--from x,y --to x,y`, positional `x,y x,y`, or legacy `--start-x/--start-y/--end-x/--end-y`)
- `gesture`
- `touch`
- `type`
- `paste` (default, `--replace`, `--file`, or `--via-menu` with a target; not `--stdin`)
- `button`
- `key`
- `key-sequence`
- `key-combo`
- `sleep <seconds>` (batch pseudo-step)

## Batch flags
- `--device <UDID>`: required simulator target.
- `--step "..."`: repeatable inline step source.
- `--file <path>`: read one step per line from file.
- `--stdin`: read one step per line from stdin.
- `--continue-on-error`: keep running after a failed step; report failures at end.
- `--ax-cache perBatch|perStep|none`: selector tap AX snapshot reuse policy. `perBatch` (default) resolves every selector against one snapshot fetched at first use; `perStep` refetches at each step; `none` never caches. `--wait-timeout` polling always refetches (and updates the cache).
- `--type-submission chunked|composite`: submission mode for `type` steps.
- `--type-chunk-size <n>`: chunk size when using chunked submission.
- `--wait-timeout <seconds>`: maximum seconds to poll for selector-based elements before failing (0 = no waiting, default).
- `--poll-interval <seconds>`: seconds between accessibility tree polls when `--wait-timeout` is active (default 0.25).
- `--verbose`: enable detailed stderr logs for troubleshooting (default quiet output).

## Input rules
- Use exactly one source: `--step` OR `--file` OR `--stdin`.
- Empty lines are ignored.
- `#` comment lines are ignored in file/stdin input.
- Do not pass `--device` or `--udid` inside step lines; keep the target at batch command level.
- A paste step runs after pending HID is flushed and uses the standalone verified clipboard and native menu path. Focus a field first, or use `--via-menu --target-id <id>` in the step. Clipboard sharing and native editing rules still apply.
- Use `paste -- 'text starting with -'` for dash-prefixed text. The option terminator does not hide the batch-level target.

## Failure and retry

- A transient boot-readiness error does not automatically replay the batch.
- Earlier steps may have completed before a later step failed. This also applies to errors collected by `--continue-on-error`.
- Inspect the current UI and completed steps before an explicit retry. Do not repeat the whole batch without checking its side effects.

## Example: inline steps
```bash
sim-use ios batch --device SIMULATOR_UDID \
--step "tap --id EmailField" \
--step "type 'cam@example.com'" \
--step "key 43" \
--step "type 'super-secret'" \
--step "key 40"
```

## Example: stdin steps
```bash
cat <<'EOF' | sim-use ios batch --device SIMULATOR_UDID --stdin
tap --id EmailField
type 'cam@example.com'
key 43
type 'super-secret'
key 40
EOF
```

## Example: file steps
`login.steps`
```text
# login flow
tap --id EmailField
type 'cam@example.com'
key 43
type 'super-secret'
key 40
```

Run:
```bash
sim-use ios batch --device SIMULATOR_UDID --file login.steps
```

## Example: explicit timing and policy
```bash
sim-use ios batch --device SIMULATOR_UDID \
--ax-cache perStep \
--type-submission chunked \
--type-chunk-size 150 \
--continue-on-error \
--step "tap --label Settings" \
--step "sleep 0.5" \
--step "tap --id SaveButton"
```

## Example: multi-screen flow with element waiting
```bash
sim-use ios batch --device SIMULATOR_UDID \
--wait-timeout 5 \
--step "tap --id LoginButton" \
--step "tap --id WelcomeMessage"
```

The second step polls for up to 5 seconds for `WelcomeMessage` to appear after the login tap triggers navigation.

If label selectors are ambiguous and sim-use reports no `AXUniqueId` values for matches, switch that step to coordinates (`tap --point x,y` or `-x/-y`).
223 changes: 223 additions & 0 deletions .agents/skills/sim-use/references/cheatsheet.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,223 @@
# sim-use Command Cheatsheet

## Command namespaces

| Namespace | Scope | Examples |
|---|---|---|
| `sim-use <verb>` | Cross-platform (iOS + Android) | `ui`, `tap`, `swipe`, `type`, `paste`, `button`, `gesture`, `screenshot`, `record-video`, `stream-video`, `app-state` |
| `sim-use ios <verb>` | iOS Simulator only | `key`, `key-combo`, `key-sequence`, `batch` |
| `sim-use android <verb>` | Android device only | `init`, `devices`, `ping` |

## Device resolution

`--device` is optional. Resolution order: `--device` flag → `$SIM_USE_DEVICE` env → only live daemon → only booted simulator.

When multiple devices exist, pass `--device <UDID>` explicitly. Run `sim-use devices` to list all connected devices across platforms.

## Selectors

```bash
sim-use tap @5 # alias from last `ui` snapshot
sim-use tap '#3' # 3rd cell of dominant list
sim-use tap '#2@2' # 2nd cell of 2nd list (quote for shell)
sim-use tap '#settingsButton' # AXUniqueId (live AX lookup)
sim-use tap --id Safari # by accessibility identifier
sim-use tap --label "General" # by exact label
sim-use tap --label-contains "Order" # substring match
sim-use tap --label-regex '^Chat.*\d+$' # ICU regex
sim-use tap -x 100 -y 200 # outline coordinates (last resort; any orientation)
sim-use tap --point 100,200 # same, pair form
sim-use tap -x 100 -y 200 --coordinate-space native # device-native portrait points
```

## Disambiguating with --frame

Narrow matches by screen region when selectors collide:

```bash
sim-use tap --label 'Tab' --frame minY=0.7r # bottom 30% of screen
sim-use tap --label 'Tab' --frame maxY=0.2r # top 20% of screen
sim-use tap --label 'OK' --frame 'minX=200,maxX=400,minY=600,maxY=700'
```

- Values: absolute pixels (`700`) or fraction with `r` suffix (`0.7r`).
- Keys: `minX`, `maxX`, `minY`, `maxY`. Each appears at most once.
- Composes with all selectors and `--element-type`.

## Text input

```bash
sim-use type 'Hello World!' # HID keyboard (ASCII)
echo "complex text" | sim-use type --stdin # stdin for special chars
sim-use paste 'こんにちは 🎉' # focused caret/selection; either iOS host
sim-use paste 'text' --replace # Select All then native Paste
sim-use paste 'text' --via-menu --target-id <id> # explicit iOS target
sim-use paste --file body.txt # UTF-8, including multiline text
```

- iOS: both modes work with software or hardware keyboards. Default paste preserves the focused field's caret and selection; explicit targeting can move them. Native editing rules apply. Verify the field and avoid concurrent shared-clipboard writers.
- Android: `paste` uses native `ACTION_PASTE`; `type` works for all unicode.

## Gestures and timing

```bash
sim-use gesture scroll-up # scroll-up = content moves up = page down
sim-use gesture scroll-down # page up
sim-use swipe --from 50,500 --to 350,500
sim-use long-press @5 # default 0.8s hold
sim-use long-press --label "Photos" # selector targeting
sim-use long-press @5 --duration 1.2 # custom hold time
sim-use tap @5 --duration 0.05 # brief hold for toggles/switches
sim-use tap @5 --pre-delay 0.3 # wait before tapping
```

`long-press` is sugar over `tap --duration 0.8`. Same targeting surface as `tap`. Use for action menus, launcher popups, drag-handle activation.

Note: some Android apps render long-press menus in a `PopupWindow` overlay that the bridge's `describe-ui` cannot walk — verify with a screenshot when this matters.

**Naming gotcha:** `scroll-up` scrolls the *content* up (like swiping up), which shows content *below* — it's "page down" in reading order.

## Pinch and rotate (two-finger gestures)

```bash
sim-use gesture pinch-out # zoom in (default scale 2.0)
sim-use gesture pinch-in # zoom out (default scale 0.5)
sim-use gesture pinch-out --scale 3.0 --radius 60 # wider zoom, smaller start circle
sim-use gesture rotate-cw # clockwise 90° (default)
sim-use gesture rotate-ccw --angle 45 # counter-clockwise 45°
sim-use gesture pinch-out --center-x 200 --center-y 400 # off-center pivot
```

| Parameter | Default | Description |
|---|---|---|
| `--scale` | 2.0 (out) / 0.5 (in) | End radius / start radius ratio |
| `--angle` | 90 | Rotation sweep in degrees |
| `--center-x/y` | screen center | Pivot point (pixels) |
| `--radius` | 80 | Start radius (pixels) |
| `--steps` | 10 | Interpolated HID Move events (iOS only) |
| `--step-ms` | derived from duration/steps | Sleep between Move events (iOS only) |

## Multi-touch (low-level)

For gestures the presets don't cover, use `multi-touch` directly with two contact points:

```bash
# Pinch-zoom in by moving two fingers apart vertically
sim-use multi-touch \
--x1 195 --y1 472 --x1-end 195 --y1-end 322 \
--x2 195 --y2 472 --x2-end 195 --y2-end 622 \
--steps 10 --step-ms 30
```

Also available: `tap --fingers 2` and `long-press --fingers 2` for two-finger tap/hold.

## Touch (low-level, single finger)

```bash
sim-use touch -x 150 -y 250 --down # touch down only
sim-use touch -x 150 -y 250 --up # touch up only
sim-use touch -x 150 -y 250 --down --up # tap
sim-use touch -x 150 -y 250 --down --up --delay 1.0 # long press
```

## Screenshots and video

```bash
sim-use screenshot --output shot.png
sim-use record-video --output recording.mp4 # H.264, 30 fps default; Ctrl+C to stop
sim-use record-video --output smooth.mp4 --fps 60 # iOS: constant rate up to 60 fps (Android ignores --fps, native rate)
sim-use record-video --output demo.gif # animated GIF (inferred from extension, or --format gif); 10 fps + 0.5 scale defaults; add --gif-markers for START/END boundary cards; transcoded after Ctrl+C
sim-use stream-video --format h264 | ffplay -f mpegts -probesize 32768 -i - # live H.264 preview in MPEG-TS (both platforms); iOS constant --fps (default 30), Android native VFR (a still screen sends no new frames)
sim-use stream-video --fps 10 --format mjpeg > out.mjpeg # screenshot-loop JPEG stream; DEPRECATED on iOS (use h264), kept on Android for devices without screenrecord
```

### Stopping a backgrounded recording

Send `SIGTERM` and `wait` — do **not** `SIGKILL`, which skips the MP4 trailer and produces an unplayable file.

```bash
sim-use record-video --output recording.mp4 &
PID=$!
# ... drive the simulator ...
kill -TERM "$PID"
wait "$PID" # blocks ~50–150 ms while the trailer is written
```

## Batch (iOS only)

```bash
sim-use ios batch --device <UDID> \
--step "tap --id SearchField" \
--step "type 'query'" \
--step "key 40" \
--wait-timeout 5
```

- One step source per run: `--step`, `--file`, or `--stdin`.
- `--wait-timeout` makes selector taps poll for the element.
- `--ax-cache perStep` refreshes the AX snapshot between steps.
- Do not put `--device` inside step lines.
- See `references/batch-reference.md` for full semantics, flags, and examples.

## Daemon

```bash
sim-use daemon status
sim-use daemon stop --device <UDID>
sim-use daemon stop --all
SIM_USE_NO_DAEMON=1 sim-use ui # bypass daemon for one call
```

Daemon is iOS-only (auto-spawned, 600s idle TTL). Android commands go through adb directly.

## iOS keyboard (HID keycodes)

```bash
sim-use ios key 40 # Enter
sim-use ios key 42 --duration 1.0 # Hold Backspace
sim-use ios key-sequence --keycodes 11,8,15,15,18 # "hello"
sim-use ios key-combo --modifiers 227 --key 4 # Cmd+A
sim-use ios key-combo --modifiers 227 --key 6 # Cmd+C
sim-use ios key-combo --modifiers 227 --key 25 # Cmd+V
sim-use ios key-combo --modifiers 227,225 --key 4 # Cmd+Shift+A
```

### Common keycodes

| Key | Code | Key | Code | Key | Code |
|---|---|---|---|---|---|
| Enter | 40 | Escape | 41 | Backspace | 42 |
| Tab | 43 | Space | 44 | a | 4 |
| LeftGUI (Cmd) | 227 | LeftShift | 225 | LeftCtrl | 224 |
| LeftAlt | 226 | F1 | 58 | F12 | 69 |

## Timing parameters

| Parameter | Range | Description | Available on |
|---|---|---|---|
| `--pre-delay` | 0–10s | Delay before action | tap, swipe, gesture |
| `--post-delay` | 0–10s | Delay after action | tap, swipe, gesture |
| `--duration` | 0–10s | Action duration | swipe, gesture, button, key, tap (opt-in for toggles) |
| `--delay` | 0–5s | Between-item delay | key-sequence, touch |

## UDID auto-resolution

`--device` is optional. Resolution order:

1. Explicit `--device <UDID>` flag
2. `$SIM_USE_DEVICE` environment variable
3. The only live sim-use daemon under `/tmp/sim-use-<uid>/` (<1 ms)
4. The only simulator with `state == "Booted"` via `xcrun simctl list` (~150 ms)

Edge cases:
- **Stale daemon**: if you shut down a simulator outside sim-use but its daemon is still alive, the next command fails loudly — run `sim-use daemon stop --all` then retry.
- **Multiple daemons + multiple booted**: falls through to step 4 and errors with a disambiguation message listing all booted devices.
- For CI fan-out (many simulators in parallel), keep `--device` explicit.

## --json envelope

Every command supports `--json`. Shape: `{ "ok": true/false, "data": {...}, "error": "...", "hint": "..." }`.

The `hint` field on errors contains actionable guidance (e.g. each match as `'label' [Type] #id` on `multipleMatches`). Use it for self-correcting retries.

For `ui --json`, prefer `data.outline` / `data.entries` / `data.lists`. The `data.raw` field is the full AX tree (~3x larger); pass `--no-raw` to omit it in agent loops (on older binaries without the flag, use `jq 'del(.data.raw)'`).
59 changes: 59 additions & 0 deletions .agents/skills/sim-use/references/crash-awareness.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Crash Awareness Protocol

sim-use automatically tracks the target app's process liveness while the daemon is running. When the tracked process disappears between commands, sim-use surfaces a banner on the next `ui` call.

## Signals

### Process liveness (iOS + Android)

```
================ PROCESS DISAPPEARED ================
com.example.app (pid 12345) was alive at the previous command and is GONE now.
Likely crash or termination, not a backgrounding. Verify before trusting subsequent actions.
=====================================================
App: SpringBoard 402x874
```

The signal is process liveness — a PID that was alive and is now gone. Backgrounding (permission dialogs, share sheets, app switcher) does **not** trip it.

### Crash dialog (Android only)

```
=============== CRASH DIALOG DETECTED ===============
An Android system app-crash dialog is on screen ("MyApp keeps stopping").
=====================================================
```

Detected directly from the accessibility tree via the AOSP framework resource IDs (`android:id/aerr_close` / `aerr_app_info`). Works without the daemon, locale-independent, and fires while the dialog is visible — even before the process fully exits.

Either signal alone is a sufficient crash hint.

## Mandatory response

1. **STOP** all automation. Do not silently relaunch or continue tapping.
2. **Report** the crash to the user, including the banner text and context (what action preceded the crash).
3. **Wait** for user instructions before proceeding.

Never assume you can recover from a crash by relaunching the app — the user may need to inspect crash logs, capture state, or change strategy.

## Confidence and idle gaps

Deaths observed after a long idle gap (default 120s, tunable with `SIM_USE_CRASH_WINDOW`) are downgraded to a quiet `[i]` line and the baseline is reset. This prevents false alarms from out-of-band kills while the agent was idle.

## Resetting the baseline

After intentionally relaunching the app or accepting a known crash:

```bash
sim-use app-state --reset --device <UDID>
```

This clears pending signals so subsequent commands don't re-surface a stale death.

## Disabling detection

```bash
SIM_USE_NO_CRASH_DETECT=1 sim-use ui --device <UDID>
```

Detection is on by default in daemon mode.
Loading
Loading