Skip to content

test(e2e): Restore iOS replay assertion in captureReplay test - #6683

Open
antonis wants to merge 1 commit into
mainfrom
test/restore-replay-e2e-assertion
Open

test(e2e): Restore iOS replay assertion in captureReplay test#6683
antonis wants to merge 1 commit into
mainfrom
test/restore-replay-e2e-assertion

Conversation

@antonis

@antonis antonis commented Sep 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

📢 Type of change

  • Bugfix

📜 Description

Re-adds the assertReplay.yml iOS check to the captureReplay e2e test, removed in #6072, and makes it deterministic instead of flaky.

The assertion was flaky for a structural reason, not CI noise: buffer-mode replay (replaysOnErrorSampleRate) only attaches a replay_id to the error event if the native replay buffer captured at least one frame before the error fired. On slow CI the exception was captured while the buffer was still empty, so the event was sent permanently without a replay_id — and no server-side retry could recover it (sentryApi.js already retries non-200s for ~10 min, so replay processing lag was never the real problem).

Changes:

  • src/EndToEndTests.tsx — add a side-effect-free Replay Ping counter (testID='replayPing') that mutates the view hierarchy without sending any events.
  • maestro/utils/primeReplayBuffer.yml (new) — tap replayPing 8× to churn the view (and spend a few seconds while Maestro re-reads the UI between taps) so the native capture records frames before the exception.
  • maestro/utils/assertReplay.yml — restored verbatim from before fix(e2e): Remove flaky iOS replay assertion from captureReplay test #6072 (asserts replay_id, duration, segments, ftypmp42 codec via the Sentry API).
  • maestro/captureReplay.yml — run the prime step, then re-wire assertReplay.yml, both gated platform: iOS.

dist/ is gitignored and rebuilt by cli.mjs on create, so the harness change flows into CI automatically.

💡 Motivation and Context

Closes #6082.

💚 How did you test it?

CI

📝 Checklist

  • I added tests to verify changes.
  • No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
  • I updated the docs if needed.
  • I updated the wizard if needed.
  • All tests passing.
  • Public API changes reviewed by another Mobile SDK team member or implemented according to the develop docs spec.
  • No breaking changes.

🔮 Next steps

#6684

@antonis @claude
test(e2e): Restore iOS replay assertion in captureReplay test
619fa90 
Re-adds the assertReplay.yml check removed in #6072. That assertion was
flaky because buffer-mode replay (replaysOnErrorSampleRate) only attaches
a replay_id to the error event if the native replay buffer captured a
frame before the error fired; on slow CI the exception was captured while
the buffer was still empty, so the event was sent permanently without a
replay_id and no server-side retry could recover it.

Make the test deterministic by priming the buffer before capturing the
exception: a new side-effect-free "Replay Ping" counter in the e2e harness
is tapped repeatedly to mutate the view hierarchy so the native capture
records frames first. iOS only, matching the assertion scope (Android does
not reliably capture replays in CI, see #4277).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@antonis antonis added the ready-to-merge Triggers the full CI test suite label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Semver Impact of This PR

⚪ None (no version bump detected)

📋 Changelog Preview

This is how your changes will appear in the changelog.
Entries from this PR are highlighted with a left border (blockquote style).

  • test(e2e): Restore iOS replay assertion in captureReplay test by antonis in #6683
  • chore(replay): Mark mobileReplayIntegration as stable by antonis in #6679
  • feat(sample): Add tvOS sample app by antonis in #6677
  • fix(e2e): Pin json gem < 3.0 for RN < 0.72 iOS builds by antonis in #6678
  • feat(ios): Expose enableMemoryIntrospection option by antonis in #6674
  • feat(android): Add anrProfilingSampleRate option by antonis in #6673
  • ci: Unpin Android E2E emulator build by antonis in #6672
  • ref(ios): Remove deprecated private SDK API usage by philprime in #6647
  • chore(deps): update Cocoa SDK to v9.27.0 by github-actions in #6670
  • chore(deps): update Sentry Android Gradle Plugin to v6.21.0 by github-actions in #6671
  • chore(deps): Bump fast-uri to ^3.1.6 to resolve security alerts by antonis in #6662
  • chore(deps): Bump qs to ^6.16.0 to resolve security alerts by antonis in #6663
  • chore(deps): Bump @xmldom/xmldom pins to patched versions to resolve security alerts by antonis in #6664
  • docs: Revamp root and nested AGENTS.md by antonis in #6666

🤖 This preview updates automatically when you update the PR.

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor
Warnings
⚠️

⚠️ Auth token handling changes detected

This PR modifies code related to Sentry auth token handling. Please ensure no auth tokens are accidentally exposed or mishandled. See GHSA-68c2-4mpx-qh95 for context.

Files with auth token changes:

  • dev-packages/e2e-tests/maestro/utils/assertReplay.yml

Generated by 🚫 dangerJS against 619fa90

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Android (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 527.08 ms 566.04 ms 38.96 ms
Size 50.56 MiB 56.46 MiB 5.90 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
57e0069+dirty 413.82 ms 453.47 ms 39.65 ms
eb93136+dirty 416.18 ms 467.32 ms 51.14 ms
9210ae6+dirty 475.41 ms 525.24 ms 49.84 ms
94af3bd+dirty 503.48 ms 542.37 ms 38.89 ms
15d4514+dirty 406.77 ms 428.06 ms 21.29 ms
5fe1c6c+dirty 401.62 ms 445.28 ms 43.66 ms
0d9949d+dirty 403.57 ms 437.00 ms 33.43 ms
7887847+dirty 416.61 ms 462.04 ms 45.43 ms
100ce80+dirty 463.66 ms 539.56 ms 75.90 ms
a3265b6+dirty 406.86 ms 449.84 ms 42.98 ms

App size

Revision Plain With Sentry Diff
57e0069+dirty 49.74 MiB 54.85 MiB 5.11 MiB
eb93136+dirty 48.30 MiB 53.58 MiB 5.28 MiB
9210ae6+dirty 48.30 MiB 53.54 MiB 5.23 MiB
94af3bd+dirty 48.30 MiB 53.57 MiB 5.26 MiB
15d4514+dirty 48.30 MiB 53.60 MiB 5.30 MiB
5fe1c6c+dirty 43.75 MiB 48.14 MiB 4.39 MiB
0d9949d+dirty 43.75 MiB 48.13 MiB 4.37 MiB
7887847+dirty 49.74 MiB 54.81 MiB 5.07 MiB
100ce80+dirty 48.30 MiB 53.46 MiB 5.15 MiB
a3265b6+dirty 48.30 MiB 53.58 MiB 5.28 MiB

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

iOS (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 3858.71 ms 1234.98 ms -2623.73 ms
Size 5.15 MiB 6.88 MiB 1.73 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
61cc206+dirty 3849.70 ms 1230.33 ms -2619.38 ms
5a21b51+dirty 3823.11 ms 1214.46 ms -2608.65 ms
038a6d7+dirty 3849.69 ms 1228.40 ms -2621.28 ms
3817909+dirty 1183.90 ms 1187.50 ms 3.60 ms
d7d54c6+dirty 3827.13 ms 1217.18 ms -2609.95 ms
0bd8916+dirty 3842.33 ms 1230.76 ms -2611.58 ms
a0d8cf8+dirty 3842.33 ms 1212.40 ms -2629.93 ms
af33f3b+dirty 3849.98 ms 1236.45 ms -2613.53 ms
5a316ea+dirty 3820.11 ms 1211.28 ms -2608.83 ms
7a89652+dirty 3861.46 ms 1229.61 ms -2631.85 ms

App size

Revision Plain With Sentry Diff
61cc206+dirty 4.98 MiB 6.55 MiB 1.57 MiB
5a21b51+dirty 5.15 MiB 6.67 MiB 1.51 MiB
038a6d7+dirty 5.15 MiB 6.70 MiB 1.55 MiB
3817909+dirty 3.38 MiB 4.73 MiB 1.35 MiB
d7d54c6+dirty 4.98 MiB 6.51 MiB 1.53 MiB
0bd8916+dirty 5.15 MiB 6.69 MiB 1.53 MiB
a0d8cf8+dirty 5.15 MiB 6.67 MiB 1.51 MiB
af33f3b+dirty 4.98 MiB 6.51 MiB 1.54 MiB
5a316ea+dirty 4.98 MiB 6.51 MiB 1.53 MiB
7a89652+dirty 5.15 MiB 6.70 MiB 1.55 MiB

@antonis
antonis marked this pull request as ready for review September 7, 2026 14:38
@antonis antonis mentioned this pull request Sep 7, 2026
Open
8 tasks
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

iOS (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 3857.73 ms 1232.65 ms -2625.08 ms
Size 5.15 MiB 6.88 MiB 1.73 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
3b6e9f9+dirty 3822.77 ms 1208.00 ms -2614.77 ms
5a23c47+dirty 3843.54 ms 1226.05 ms -2617.50 ms
100ce80+dirty 3843.57 ms 1226.46 ms -2617.12 ms
0bd8916+dirty 3816.17 ms 1215.05 ms -2601.12 ms
a0d8cf8+dirty 3826.15 ms 1213.12 ms -2613.03 ms
0a9e622+dirty 3825.35 ms 1219.04 ms -2606.31 ms
21a1e70+dirty 3826.22 ms 1217.74 ms -2608.47 ms
9ad4522+dirty 3862.28 ms 1226.27 ms -2636.01 ms
88735e9+dirty 3837.80 ms 1216.39 ms -2621.41 ms
d0e3b3e+dirty 3846.64 ms 1226.27 ms -2620.38 ms

App size

Revision Plain With Sentry Diff
3b6e9f9+dirty 5.15 MiB 6.68 MiB 1.53 MiB
5a23c47+dirty 4.98 MiB 6.46 MiB 1.49 MiB
100ce80+dirty 5.15 MiB 6.67 MiB 1.51 MiB
0bd8916+dirty 5.15 MiB 6.69 MiB 1.53 MiB
a0d8cf8+dirty 5.15 MiB 6.67 MiB 1.51 MiB
0a9e622+dirty 4.98 MiB 6.51 MiB 1.53 MiB
21a1e70+dirty 4.98 MiB 6.46 MiB 1.49 MiB
9ad4522+dirty 4.98 MiB 6.56 MiB 1.58 MiB
88735e9+dirty 4.98 MiB 6.46 MiB 1.49 MiB
d0e3b3e+dirty 4.98 MiB 6.51 MiB 1.53 MiB

@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Android (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 441.42 ms 462.96 ms 21.54 ms
Size 50.56 MiB 56.46 MiB 5.90 MiB

Baseline results on branch: main

Startup times

Revision Plain With Sentry Diff
eb93136+dirty 500.37 ms 532.58 ms 32.21 ms
9210ae6+dirty 444.14 ms 459.80 ms 15.66 ms
94af3bd+dirty 413.04 ms 451.76 ms 38.71 ms
7d6fd3a+dirty 435.06 ms 458.78 ms 23.72 ms
15d4514+dirty 413.63 ms 449.62 ms 35.99 ms
9ad4522+dirty 432.43 ms 450.91 ms 18.49 ms
0a147b2+dirty 442.80 ms 522.24 ms 79.44 ms
7887847+dirty 420.47 ms 460.55 ms 40.08 ms
a3265b6+dirty 410.96 ms 444.76 ms 33.80 ms
57e0069+dirty 442.25 ms 486.64 ms 44.39 ms

App size

Revision Plain With Sentry Diff
eb93136+dirty 48.30 MiB 53.58 MiB 5.28 MiB
9210ae6+dirty 48.30 MiB 53.54 MiB 5.23 MiB
94af3bd+dirty 48.30 MiB 53.57 MiB 5.26 MiB
7d6fd3a+dirty 43.94 MiB 49.00 MiB 5.06 MiB
15d4514+dirty 48.30 MiB 53.60 MiB 5.30 MiB
9ad4522+dirty 49.74 MiB 55.38 MiB 5.63 MiB
0a147b2+dirty 49.74 MiB 55.08 MiB 5.34 MiB
7887847+dirty 49.74 MiB 54.81 MiB 5.07 MiB
a3265b6+dirty 48.30 MiB 53.58 MiB 5.28 MiB
57e0069+dirty 49.74 MiB 54.85 MiB 5.11 MiB

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alwx alwx Awaiting requested review from alwx alwx is a code owner
@lucas-zimerman lucas-zimerman Awaiting requested review from lucas-zimerman lucas-zimerman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

ready-to-merge Triggers the full CI test suite

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add back replay assertion from captureReplay E2E test

1 participant

@antonis