Skip to content

feat(html-reporter): write the run as one HTML page - #1642

Merged
okwasniewski merged 4 commits into
mainfrom
krzysztof/html-reporter
Nov 10, 2026
Merged

okwasniewski merged 4 commits into
mainfrom
krzysztof/html-reporter

Conversation

@KrzysztofMoch

@KrzysztofMoch KrzysztofMoch commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

e2e reports a run as terminal output, JSON, JUnit, markdown, or a pull request comment. None of them shows a test's steps next to its screenshots and video, and the Playwright and Maestro migration guides list reporter: 'html' as missing.

What changed

  • New package @e2e-dev/html-reporter. htmlReporter() writes <output>/report.html, one page that opens from disk with no server:

    import { htmlReporter } from '@e2e-dev/html-reporter';
    
    export default { targets, reporters: ['list', htmlReporter()] } satisfies E2EConfig;
  • The page is a React app in packages/html-reporter/app/. Vite and vite-plugin-singlefile build it into one HTML file with the fonts and icons inlined. The reporter (packages/html-reporter/src/reporter.ts) fills that template with the report-1 document, the media, and the test-file lines around each step.

  • media picks how screenshots and videos reach the page: 'link' (default, relative paths into results/), 'embed' (one file to send), or 'none'.

  • The page shows the tests as a tree (file, describe blocks, tests, failures first) with status tabs and search. Each test shows its attempts, the error with expected and received values diffed, the code frame, its steps synced to the video, agent step details, and a copy prompt button that puts the failure on the clipboard for a coding agent.

  • Core: FinishedRun.redact(text) (packages/e2e/src/types.ts). It masks configured secret values in text a reporter brings from outside the report, built from the same ledger as TestFilterContext.redact. The page uses it on the test-file lines it embeds. This needs the next e2e release, so the reporter's e2e peer floor is >=0.23.1.

  • media: 'embed' leaves out downloads the runner could not scan for secrets (redaction: 'incomplete'), and download links save the file instead of opening it.

  • Fonts are Geist and JetBrains Mono (OFL 1.1) and the icons are Lucide (ISC). The licence texts ship in the package and are listed in NOTICE.

flowchart LR
  A[run finished] --> B["writeReport(run)"]
  B --> C[report-1 document]
  B --> D["media: link / embed / none"]
  B --> E["source lines, run.redact"]
  C & D & E --> F["dist/app/index.html template"]
  F --> G[".e2e/report.html"]
Loading

Not covered yet (documented in docs/reference/reporters.mdx): after --last-failed the page shows only the rerun's tests, and under e2e explore it shows no findings.

Verified

Ran it locally: yes

  • Testbed, built CLI, with htmlReporter() on the todos and forms suites plus one deliberately failing test, run with --repeat-each 2:

    HTML report  .e2e-html/report-link.html
    
         Failed  × |web| tests-html-scratch/broken.e2e.ts:10 > scratch > fails on a wrong remaining count
                   ASSERTION_FAILED: expect.toHaveText failed
                 × |web| tests-html-scratch/broken.e2e.ts:5 > scratch > fails on a wrong remaining count (repeat #1)
    
  • The testbed's reporter stress suite (hostile titles, a 20 KiB message, timeouts, flakes, skips), plus a test that fills the configured admin password and then fails, with media: 'embed'. Every one of the 16 test pages opened in Chromium with no page errors and no dialogs. The password value is nowhere in report.html, and the frame shows fill('<secret:admin.password>').

  • Pages from the web benchmark agent suite and the mobile benchmark on an iOS simulator, in both themes and at phone width.

Run list Failed test
run list failed test
Light theme Reporter stress suite Phone recording
light stress mobile player

Review

  • A fresh-context review found 9 issues. Fixed: unscanned downloads embedded in a page meant for sending, --repeat-each rows that looked identical, download links that opened in the report's origin, emoji splitting in the expected/received diff, and no runner-level test for run.redact (packages/e2e/tests/integration/reporters.test.ts, which fails when redact passes text through).

  • Declined:

    • Folding --last-failed and explore findings into the page: documented above as a follow-up.
    • Streaming 'embed' past V8's string limit (about 400 MB of media): the docs already warn that device video makes an embedded page large.
    • A test against the built template: the root test script builds first, and embedPayload already throws when the placeholder is not there exactly once.
  • Cubic: fixed 15 threads (copied-prompt fence, cache-only step details, headline counted from the listed tests, the reporter abort signal, video part clock, serial member error fallback, a doubled screen-reader status, and docs wording). Two threads about the secrets invariant are left open for a human: symlinked artifacts, and multiline secrets in code frames.

Reviewed with sol-loop: 2 rounds with gpt-6.1-sol. Sol verdict: approved (before the fresh-context fixes above).

Closes ENG-1079.

Written with Claude Code (claude-opus-5-5).

View guided diff Turn on auto-fix

@github-actions github-actions Bot added the size:XXL 1,000+ changed lines, not counting tests, fixtures, translated docs, or the lockfile label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web: 91 passed

All 91 tests in 41 files
Test Time
🟢 tests/aria-hidden-flow.e2e.ts · 2 passed 629ms
🟢 aria hidden flow › picks the Pro plan by its visible text and confirms it 478ms
🟢 aria hidden flow › another plan shows the error and keeps the picker open 151ms
🟢 tests/async-states.e2e.ts · 2 passed 7.1s
🟢 async states › reads the code off the transient toast and verifies it once the button enables 2.3s
🟢 async states › a wrong code is rejected after the toast has gone 4.8s
🟢 tests/canvas-only.e2e.ts · 2 passed 272ms
🟢 canvas only › taps square, circle, then diamond at their pixel coordinates 142ms
🟢 canvas only › a shape out of order paints the error and resets progress 130ms
🟢 tests/canvas-whiteboard.e2e.ts · 2 passed 384ms
🟢 canvas whiteboard › drags both shapes into their slots and saves 229ms
🟢 canvas whiteboard › bounces a shape off the wrong slot and refuses to save early 155ms
🟢 tests/cart-totals.e2e.ts · 2 passed 358ms
🟢 cart totals › increasing a quantity updates the line total but the order summary stays stale (planted bug) 221ms
🟢 cart totals › a quantity never drops below one 137ms
🟢 tests/checkout-review.e2e.ts · 1 passed 145ms
🟢 shows a raw greeting token and an undefined delivery date (planted bug), yet places the order 145ms
🟢 tests/context-menu-trap.e2e.ts · 2 passed 531ms
🟢 context menu trap › right-click opens nothing; the row menu renames the file 256ms
🟢 context menu trap › the decoy menu items refuse and keep the file 275ms
🟢 tests/control-inventory.e2e.ts · 8 passed 3.6s
🟢 control inventory › check sets the checkbox and the radio without flipping a set one 251ms
🟢 control inventory › check picks a radio the pick replaces with its summary 156ms
🟢 control inventory › a long press and a double click are told apart from a tap 995ms
🟢 control inventory › the file menu opens on right-click only and Rename renames the file 249ms
🟢 control inventory › setInputFiles attaches files named from the project root 170ms
🟢 control inventory › the details view is left through the browser history 180ms
🟢 control inventory › scrollIntoView brings the footnote into view 285ms
🟢 control inventory › every exercise done shows the success message 1.3s
🟢 tests/css-content-ui.e2e.ts · 2 passed 315ms
🟢 css content ui › the tile labelled Ship it advances and confirming ships 177ms
🟢 css content ui › a wrong tile shows the CSS-rendered error and keeps the picker 138ms
🟢 tests/date-picker.e2e.ts · 2 passed 544ms
🟢 date picker › navigates three months ahead and books the required date 310ms
🟢 date picker › confirming without the exact date shows the error 234ms
🟢 tests/debounced-search.e2e.ts · 2 passed 2.8s
🟢 debounced search › a decoy one character off is refused 1.4s
🟢 debounced search › picking the exact item adds it to the cart 1.4s
🟢 tests/deferred-dialog.e2e.ts · 1 passed 1.1s
🟢 one submit opens one confirm, even when double-clicked 1.1s
🟢 tests/div-soup.e2e.ts · 2 passed 615ms
🟢 div soup › a wrong code is rejected 267ms
🟢 div soup › the right code grants access after the delete tile fixes a typo 348ms
🟢 tests/drag-and-drop.e2e.ts · 2 passed 682ms
🟢 drag and drop › an out-of-order item resets the dropzone 319ms
🟢 drag and drop › dropping Banana then Cherry passes 363ms
🟢 tests/filter-deep-link.e2e.ts · 2 passed 246ms
🟢 filter deep link › filtering writes the query to the URL 140ms
🟢 filter deep link › opening a shared link restores the filter 106ms
🟢 tests/gift-card-purchase.e2e.ts · 3 passed 585ms
🟢 gift card purchase › refuses to buy without an amount 157ms
🟢 gift card purchase › refuses a recipient without an at sign 210ms
🟢 gift card purchase › sends the gift card 218ms
🟢 tests/hover-menu.e2e.ts · 2 passed 443ms
🟢 hover menu › a decoy leaf is refused 196ms
🟢 hover menu › hovering Account, then Billing, reaches Redeem voucher 247ms
🟢 tests/iframe-form.e2e.ts · 2 passed 495ms
🟢 iframe form › a wrong code is refused by the host page 245ms
🟢 iframe form › the code read from one frame applies through the other 250ms
🟢 tests/image-only-ui.e2e.ts · 3 passed 502ms
🟢 image only ui › the icons have no names, so the bare svg locator is ambiguous 110ms
🟢 image only ui › a wrong icon shows the error image and keeps the icons 183ms
🟢 image only ui › tapping the star, then the heart, verifies the icons 209ms
🟢 tests/infinite-scroll.e2e.ts · 1 passed 2.7s
🟢 scrolls the feed until the Golden Ticket loads, then claims it 2.7s
🟢 tests/kanban-board.e2e.ts · 3 passed 687ms
🟢 kanban board › pointer drags move both cards into the goal layout 370ms
🟢 kanban board › a swipe along a path drags a card as well 148ms
🟢 kanban board › a plain tap moves nothing and the untouched board is rejected 169ms
🟢 tests/login-form.e2e.ts · 4 passed 693ms
🟢 login form › rejects a malformed email 161ms
🟢 login form › rejects a short password 165ms
🟢 login form › rejects an unknown account 168ms
🟢 login form › signs in and logs out 199ms
🟢 tests/lying-labels.e2e.ts · 4 passed 599ms
🟢 lying labels › the button whose accessible name says Pay now is the wrong one 135ms
🟢 lying labels › a text match for Pay now catches the decoy phrase too 101ms
🟢 lying labels › the field whose accessible name says Amount is the wrong one 178ms
🟢 lying labels › visible text drives the payment through 185ms
🟢 tests/native-dialogs.e2e.ts · 3 passed 433ms
🟢 native dialogs › answers the confirm, prompt, and alert chain 144ms
🟢 native dialogs › dismissing the confirm cancels the order 140ms
🟢 native dialogs › a wrong gift code is rejected 149ms
🟢 tests/newsletter-signup.e2e.ts · 1 passed 2.2s
🟢 subscribe renders no feedback and logs a TypeError (planted bug) 2.2s
🟢 tests/onboarding-wizard.e2e.ts · 5 passed 4.4s
🟢 onboarding wizard › back keeps what was typed and the brief comes back with it 262ms
🟢 onboarding wizard › the address step takes the brief through its bare-div dropdown 314ms
🟢 onboarding wizard › the profile step steers the calendar three decades back 1.1s
🟢 onboarding wizard › the preferences step flips the toggles the brief names and leaves the trap off 1.3s
🟢 onboarding wizard › the review step repeats the brief and finishes with the PIN 1.4s
🟢 tests/order-history.e2e.ts · 2 passed 1.4s
🟢 order history › overview and settings render their panels 194ms
🟢 order history › the orders tab never leaves its loading state (planted bug) 1.3s
🟢 tests/otp-auto-advance.e2e.ts · 3 passed 1.4s
🟢 otp auto-advance › focus advances with each digit and the full code verifies 485ms
🟢 otp auto-advance › backspace on an empty box clears and refocuses the previous one 143ms
🟢 otp auto-advance › a wrong code clears every box and starts over 803ms
🟢 tests/overlay-trap.e2e.ts · 1 passed 716ms
🟢 continues only after the banner and the overlay are gone 716ms
🟢 tests/playbook-cleanup.e2e.ts · 2 passed 513ms
🟢 playbook cleanup › deleting the draft removes it from an exact-name search 266ms
🟢 playbook cleanup › cancelling the dialog keeps the draft searchable 247ms
🟢 tests/price-sorting.e2e.ts · 3 passed 398ms
🟢 price sorting › opens in the featured order with sale prices struck through 109ms
🟢 price sorting › low to high sorts by the original price, not the paid one (planted bug) 142ms
🟢 price sorting › high to low is the same wrong order reversed (planted bug) 147ms
🟢 tests/product-reviews.e2e.ts · 1 passed 1.1s
🟢 the running-shoe page lists coffee-grinder reviews (planted bug) 1.1s
🟢 tests/promo-storefront.e2e.ts · 1 passed 1.3s
🟢 buys a deal once the promo chrome is out of the way 1.3s
🟢 tests/rich-text-editor.e2e.ts · 2 passed 495ms
🟢 rich text editor › bolding exactly the required word passes the check 243ms
🟢 rich text editor › bold that spills into the previous word fails the check 252ms
🟢 tests/saved-addresses.e2e.ts · 3 passed 631ms
🟢 saved addresses › adds an address and clears the form 206ms
🟢 saved addresses › ignores an add with an empty field 193ms
🟢 saved addresses › delete shows its toast but keeps the address, on retry too (planted bug) 232ms
🟢 tests/shadow-dom-form.e2e.ts · 2 passed 580ms
🟢 shadow DOM form › rejects a wrong access code 282ms
🟢 shadow DOM form › grants access with the code from the hint inside the root 298ms
🟢 tests/smoke.e2e.ts · 1 passed 381ms
🟢 home lists the scenarios and opens one 381ms
🟢 tests/stale-dom.e2e.ts · 1 passed 2.2s
🟢 taps the moving target three times across rebuilds of the list 2.2s
🟢 tests/sticky-chrome.e2e.ts · 1 passed 172ms
🟢 accepts the terms once the button clears the sticky footer 172ms
🟢 tests/virtualized-table.e2e.ts · 1 passed 14.9s
🟢 scrolls the windowed table to the Golden Row and claims it 14.9s
🟢 tests/wishlist.e2e.ts · 2 passed 1.5s
🟢 wishlist › saving toggles the button and the tab badge 221ms
🟢 wishlist › the wishlist tab renders the empty state despite the badge count (planted bug) 1.3s

Tested by e2e 0.23.0 · 1m 1s · web · run artifacts

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e web agent: 33 passed

34 agent steps · 26 replayed from cache · 20 model calls · 88.3k tokens (94% cached) · $0.0035

All 33 tests in 5 files
Test Agent Time
🟢 tests-agent/control-inventory.e2e.ts · 7 passed 7 steps 7.8s
🟢 control inventory › check sets the checkbox and the radio 1 step 839ms
🟢 control inventory › check picks a radio the pick replaces with its summary 1 step 292ms
🟢 control inventory › a long press and a double click are chosen over a plain tap 1 step 904ms
🟢 control inventory › a right-click opens the file menu and Rename renames the file 1 step 2.6s
🟢 control inventory › upload attaches the named project files 1 step 2.5s
🟢 control inventory › back returns from the details view to the inventory 1 step 301ms
🟢 control inventory › scroll_to brings the footnote into view 1 step 378ms
🟢 tests-agent/login-form.e2e.ts · 1 passed 1 step 396ms
🟢 act signs in with the benchmark account 1 step 396ms
🟢 tests-agent/planted-bugs.e2e.ts · 3 passed 4 steps · 8 calls 13.8s
🟢 cart totals: a judgment catches the stale order summary 2 steps · 4 calls 6.2s
🟢 checkout review: a judgment catches the broken copy 1 step · 1 call 2.6s
🟢 wishlist: a judgment catches the lost saved items 1 step · 3 calls 5.0s
🟢 tests-agent/scenarios.e2e.ts · 21 passed 21 steps · 12 calls 1m 6s
🟢 act completes shadow-dom-form 1 step · 3 calls 4.4s
🟢 act completes lying-labels 1 step · 3 calls 5.1s
🟢 act completes iframe-form 1 step · 3 calls 4.5s
🟢 act completes infinite-scroll 1 step 3.4s
🟢 act completes virtualized-table 1 step 16.4s
🟢 act completes overlay-trap 1 step 848ms
🟢 act completes sticky-chrome 1 step 906ms
🟢 act completes async-states 1 step · 3 calls 7.0s
🟢 act completes debounced-search 1 step 2.3s
🟢 act completes hover-menu 1 step 498ms
🟢 act completes drag-and-drop 1 step 454ms
🟢 act completes kanban-board 1 step 657ms
🟢 act completes context-menu-trap 1 step 2.6s
🟢 act completes stale-dom 1 step 432ms
🟢 act completes otp-auto-advance 1 step 655ms
🟢 act completes date-picker 1 step 745ms
🟢 act completes rich-text-editor 1 step 9.9s
🟢 act completes playbook-cleanup 1 step 525ms
🟢 act completes promo-storefront 1 step 2.3s
🟢 act completes gift-card-purchase 1 step 2.5s
🟢 act filters the list and the URL carries the filter 1 step 252ms
🟢 tests-agent/unique-cache.e2e.ts · 1 passed 1 step 305ms
🟢 act adds an address with a run-unique label 1 step 305ms

Tested by e2e 0.23.0 · 1m 29s · web · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 issues found across 48 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/html-reporter/src/reporter.ts">

<violation number="1" location="packages/html-reporter/src/reporter.ts:112">
P1: The containment check is lexical, but `readFile` follows symlinks; an artifact inside `artifactsRoot` can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in `sourceFrames`.</violation>

<violation number="2" location="packages/html-reporter/src/reporter.ts:153">
P1: Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into `report.html`. Redact the complete frame text before splitting it into lines.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

const file = path.resolve(artifactsRoot, artifactPath);
const inside = path.relative(artifactsRoot, file);
if (inside === '' || inside.startsWith('..') || path.isAbsolute(inside)) continue;
const bytes = await readFile(file).catch((error: unknown) => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: The containment check is lexical, but readFile follows symlinks; an artifact inside artifactsRoot can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in sourceFrames.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/html-reporter/src/reporter.ts, line 112:

<comment>The containment check is lexical, but `readFile` follows symlinks; an artifact inside `artifactsRoot` can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in `sourceFrames`.</comment>

<file context>
@@ -0,0 +1,182 @@
+    const file = path.resolve(artifactsRoot, artifactPath);
+    const inside = path.relative(artifactsRoot, file);
+    if (inside === '' || inside.startsWith('..') || path.isAbsolute(inside)) continue;
+    const bytes = await readFile(file).catch((error: unknown) => {
+      if (error instanceof Error && 'code' in error && error.code === 'ENOENT') return undefined;
+      throw error;
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] responding on behalf of KrzysztofMoch

Not changing this here; leaving it for the owner since it touches the secrets invariant. Every path under results/ is written by the runner, and the only way to plant a symlink there is test, config, or engine code, which AGENTS.md says already runs with the runner's full OS authority: it can read any file and write it into the report itself. The lexical check stops a report path from walking out of the results directory, which is the case a report can actually cause. sourceFrames reads lines of the project's own test files, then redacts them.

frames[file] = Object.fromEntries(
[...lines].flatMap((line) => {
const content = all[line - 1];
return content === undefined ? [] : [[String(line), redact(content).slice(0, MAX_LINE_CHARS)]];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into report.html. Redact the complete frame text before splitting it into lines.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/html-reporter/src/reporter.ts, line 153:

<comment>Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into `report.html`. Redact the complete frame text before splitting it into lines.</comment>

<file context>
@@ -0,0 +1,182 @@
+    frames[file] = Object.fromEntries(
+      [...lines].flatMap((line) => {
+        const content = all[line - 1];
+        return content === undefined ? [] : [[String(line), redact(content).slice(0, MAX_LINE_CHARS)]];
+      }),
+    );
</file context>

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-opus-5-5] responding on behalf of KrzysztofMoch

Leaving this for the owner too. Each line goes through redactFragments, which also masks any run of 8 or more characters of a secret, so every line of a multiline value that carries 8+ characters of it is masked. Redacting the whole file first would replace a multiline value with one <secret:...> token and shift every later line number, so frames would point at the wrong code. What can remain is a line holding under 8 characters of a multiline secret.

Comment thread packages/html-reporter/app/prompt.ts Resolved
Comment thread packages/html-reporter/app/test.tsx Resolved
Comment thread packages/html-reporter/app/list.tsx Resolved
Comment thread .changeset/html-reporter.md Resolved
Comment thread .changeset/html-reporter.md Resolved
Comment thread packages/html-reporter/app/data.ts Resolved
Comment thread packages/html-reporter/app/ui.tsx Resolved
Comment thread docs/reference/reporters.mdx Resolved
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android agent: 20 passed

24 agent steps · 15 replayed from cache · 42 model calls · 240.8k tokens (83% cached) · $0.0156

All 20 tests in 1 file
Test Agent Time
🟢 tests-agent/scenarios.e2e.ts · 20 passed 24 steps · 42 calls 7m 19s
🟢 act completes Login Form 1 step 39.1s
🟢 act completes Infinite Scroll List 1 step 34.2s
🟢 act completes Modal Flow 1 step 16.9s
🟢 act completes Bottom Tabs 1 step 11.3s
🟢 act completes Text Input Variations 1 step 8.5s
🟢 act completes Broken Accessibility 2 steps · 6 calls 23.6s
🟢 act completes Vision Only 2 steps · 1 call 7.2s
🟢 act completes Gestures 1 step · 7 calls 21.2s
🟢 act completes Async States 1 step · 15 calls 42.6s
🟢 act completes Debounced Search 1 step 7.8s
🟢 act completes Huge Virtualized List 1 step 2m 1s
🟢 act completes Flattened Registration Form 2 steps · 1 call 9.1s
🟢 act completes Flattened Login 2 steps · 6 calls 18.7s
🟢 act completes Sticky Chrome Target 1 step 11.2s
🟢 act completes WebView Accessibility 1 step · 6 calls 18.6s
🟢 act completes Permission Prompt 1 step 7.2s
🟢 act completes Product Catalog 1 step 10.0s
🟢 act completes Error Recovery 1 step 10.3s
🟢 act completes Choice Controls 1 step 9.2s
🟢 act completes Sequential Onboarding 1 step 11.3s

Tested by e2e 0.23.0 · 7m 20s · android-emulator · run artifacts

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios: 55 passed, 1 skipped

All 56 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 23 passed, 1 skipped 2m 56s
🟢 control inventory › a label query resolves a labeled text field 11.2s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 16.4s
🟢 control inventory › a placeholder query and attribute reads find the field 5.5s
🟢 control inventory › a secure field takes a fill and denies value reads 8.8s
🟢 control inventory › press sends a character and Enter through the soft keyboard 9.1s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 9.9s
🟢 control inventory › toBeFocused follows the focus 8.8s
🟢 control inventory › a switch checks, unchecks, and reports its state 9.8s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 8.4s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 12.3s
🟢 control inventory › a radio group reports the checked option 8.6s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 7.3s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.2s
🟢 control inventory › a header answers its accessible name 4.2s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 6.2s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 6.8s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.2s
🟢 control inventory › pointer verbs land at the point asked for 9.8s
🟢 control inventory › an ambiguous locator fails at once 5.5s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.5s
🟢 refusals › verbs a device never declares are refused before the engine 2ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 503ms
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 11 passed 1m 31s
🟢 device fixture › setAppearance flips the color scheme the app reads 11.5s
🟢 device fixture › setOrientation rotates the window the app measures 10.0s
🟢 device fixture › the clipboard reads back what was written 5.9s
🟢 device fixture › home leaves the app, openApp brings it back where it was 7.6s
🟢 device fixture › closeApp then openApp relaunches on the home list 7.7s
🟢 device fixture › closeApp leaves the next commands on the same device 11.0s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 6.4s
🟢 device fixture › device.back pops the scenario 5.5s
🟢 device fixture › unstable_raw reaches the test session through agent-device 4.2s
🟢 device fixture › setLocation and clearLocation change what the app reads 10.4s
🟢 device fixture › enrollBiometrics and setBiometrics answer a biometric prompt 10.7s
🟢 tests/device.e2e.ts · 3 passed 1m 5s
🟢 sequential onboarding echoes the exact values 40.4s
🟢 onboarding rejects a malformed email before advancing 13.7s
🟢 dismissing the keyboard over choice controls leaves them as they were 11.3s
🟢 tests/flows-lists.e2e.ts · 7 passed 4m 17s
🟢 infinite scroll reaches the target item 35.4s
🟢 product catalog adds exactly two of the right variant 10.3s
🟢 gestures: long-press, then swipe left 9.2s
🟢 sticky chrome: accept below the fold, then continue 11.2s
🟢 async states: load, pull to refresh, claim through the toast 10.8s
🟢 huge virtualized list reaches row 512 2m 46s
🟢 web view coupon form applies the code shown on the page 13.5s
🟢 tests/flows.e2e.ts · 7 passed 1m 10s
🟢 text inputs unlock the submit button 13.1s
🟢 filled text inputs report their values 10.8s
🟢 modal flow confirms through the native alert 9.8s
🟢 bottom tabs act inside the last tab 7.6s
🟢 error recovery retries, then confirms the delete 9.4s
🟢 choice controls place the exact order 7.5s
🟢 debounced search selects the target once results arrive 12.1s
🟢 tests/login-form.e2e.ts · 3 passed 48.8s
🟢 login form › rejects a malformed email 14.1s
🟢 login form › rejects an unknown account 14.2s
🟢 login form › signs in and logs out 20.5s
🟢 tests/smoke.e2e.ts · 1 passed 8.0s
🟢 home lists the scenarios and opens one 8.0s

Tested by e2e 0.23.0 · 6m 53s · ios-simulator · run artifacts

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped

25 agent steps · 12 replayed from cache · 59 model calls · 429.2k tokens (81% cached) · $0.0305

⚠️ 1 flaky test passed on a retry

⚠️ act completes Stripe PaymentSheet
tests-agent/scenarios.e2e.ts:266

ACTION_FAILED at step 4 of 4: agent.act "Complete this scenario as the screen instructs: open the checkout and pay with the Stripe test card 4242 4242 4242 4242, expiry 12/34, CVC 123", after 28.5s and 8 model calls

agent.act failed: Entered the test card 4242 4242 4242 4242, expiry 12/34, CVC 123, and postal code SW1A1AA; the Pay $10.99 button is enabled, but payment was not submitted and no completion outcome is shown.

  • Turn 5: type({"target":"n1275","value":"123"}) → [type] type #n1275 failed: LOCATOR_NOT_FOUND: node #n1275 is not on the current screen (observation b9); it was removed or never existed
  • Turn 6: type({"target":"n12995","value":"123"}) → [type] type #n12995 failed: APP_UNREACHABLE: perform fill failed: text entry read-back did not match the requested text Hint: The app may have formatted, filte…
  • Turn 7: type({"target":"n13206","value":"SW1A 1AA"}) → [type] type #n13206 failed: APP_UNREACHABLE: perform fill failed: text entry read-back did not match the requested text Hint: The app may have formatted, filte…
  • Screen: dev.e2e.benchmark / Stripe PaymentSheet

Evidence: screenshot, log

All 22 tests in 1 file
Test Agent Time
⚠️ tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped 25 steps · 59 calls 9m 0s
🟢 act completes Login Form 1 step 40.0s
🟢 act completes Infinite Scroll List 1 step 36.5s
🟢 act completes Modal Flow 1 step 10.0s
🟢 act completes Bottom Tabs 1 step 7.4s
🟢 act completes Text Input Variations 1 step 14.2s
🟢 act completes Broken Accessibility 2 steps · 9 calls 47.2s
🟢 act completes Vision Only 2 steps · 6 calls 31.5s
🟢 act completes Gestures 1 step · 7 calls 32.1s
🟢 act completes Async States 1 step · 10 calls 35.8s
🟢 act completes Debounced Search 1 step · 1 call 27.8s
🟢 act completes Flattened Registration Form 2 steps · 1 call 29.3s
🟢 act completes Flattened Login 2 steps · 10 calls 58.5s
🟢 act completes Sticky Chrome Target 1 step 14.3s
🟢 act completes WebView Accessibility 1 step 12.7s
🟢 act completes Permission Prompt 1 step 11.7s
🟢 act completes Photo Picker 1 step · 2 calls 31.3s
🟢 act completes Product Catalog 1 step 10.1s
🟢 act completes Error Recovery 1 step 9.4s
🟢 act completes Choice Controls 1 step · 5 calls 36.6s
⚠️ act completes Stripe PaymentSheet (1 failed attempt first) 1 step · 8 calls 32.1s
🟢 act completes Apple Pay 1 step 11.8s
⏭️ act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there)

Tested by e2e 0.23.0 · 9m 35s · ios-simulator · run artifacts

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 12 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread .changeset/html-reporter.md Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

🟢 e2e android: 57 passed, 1 skipped

All 58 tests in 7 files
Test Time
⏭️ tests/controls.e2e.ts · 24 passed, 1 skipped 3m 31s
🟢 control inventory › a label query resolves a labeled text field 32.6s
🟢 control inventory › a labeled field answers name, display value, value reads, fill, and clear 15.9s
🟢 control inventory › a placeholder query and attribute reads find the field 8.4s
🟢 control inventory › a secure field takes a fill and denies value reads 10.6s
🟢 control inventory › press sends a character and Enter through the soft keyboard 12.8s
🟢 control inventory › a field autofocuses on mount and focus() moves the focus 11.5s
🟢 control inventory › toBeFocused follows the focus 10.2s
🟢 control inventory › a switch checks, unchecks, and reports its state 10.5s
🟢 control inventory › checkbox and radio roles answer getByRole, and a checkbox toggles by tap 7.9s
🟢 control inventory › checkbox and radio roles check, uncheck, and report their state 10.3s
🟢 control inventory › a tab role answers getByRole 4.5s
🟢 control inventory › a radio group reports the checked option 8.3s
🟢 control inventory › buttons report disabled, enabled, and a state that arrives later 8.6s
🟢 control inventory › a display: none element is absent rather than hidden, its visible twin attached 7.4s
🟢 control inventory › a header answers its accessible name 4.1s
🟢 control inventory › a tab row answers count, all, texts, filters, positions, and switching 7.2s
🟢 control inventory › a list answers counts, list-form text, reads, and boxes in order 7.1s
⏭️ control inventory › a row contains its texts (skipped: React Native flattens a plain View out of the accessibility tree on both platforms: XCTest and the Android helper report a View with a testID as a leaf beside its children, so filter({ has }) and a query scoped to it match nothing; only Sc…)
🟢 control inventory › a long press lands as one 8.1s
🟢 control inventory › a double tap lands inside the 300 ms window 6.8s
🟢 control inventory › an ambiguous locator fails at once 5.7s
🟢 control inventory › screenshot, restart, and clearState reopen the app on its home list 10.0s
🟢 refusals › verbs a device never declares are refused before the engine 6ms
🟢 refusals › keys the soft keyboard cannot send are refused by the engine 1.6s
🟢 refusals › a missing locator fails as not found once its timeout passes 1.1s
🟢 tests/device-fixture.e2e.ts · 10 passed 1m 40s
🟢 device fixture › setOrientation rotates the window the app measures 6.6s
🟢 device fixture › home leaves the app, openApp brings it back where it was 7.0s
🟢 device fixture › foregroundApp sees the home screen after home() 5.4s
🟢 device fixture › closeApp then openApp relaunches on the home list 9.4s
🟢 device fixture › closeApp leaves the next commands on the same device 9.9s
🟢 device fixture › foregroundApp after closeApp reports the closed session as APP_NOT_OPEN 4.8s
🟢 device fixture › device.back pops the scenario 5.8s
🟢 device fixture › unstable_raw reaches the test session through agent-device 5.0s
🟢 device fixture › setNetwork and setAirplaneMode change what the app sees 17.4s
🟢 device fixture › setLocation and clearLocation change what the app reads 28.5s
🟢 tests/device.e2e.ts · 5 passed 54.6s
🟢 permission prompt › allowing the system dialog enables the microphone 9.3s
🟢 permission prompt › a denied prompt recovers once the permission is granted outside 18.4s
🟢 sequential onboarding echoes the exact values 12.3s
🟢 onboarding rejects a malformed email before advancing 7.1s
🟢 dismissing the keyboard over choice controls leaves them as they were 7.5s
🟢 tests/flows-lists.e2e.ts · 7 passed 3m 25s
🟢 infinite scroll reaches the target item 30.5s
🟢 product catalog adds exactly two of the right variant 8.7s
🟢 gestures: long-press, then swipe left 8.3s
🟢 gestures: double-tap completes the flow 11.0s
🟢 sticky chrome: accept below the fold, then continue 12.2s
🟢 async states: load, pull to refresh, claim through the toast 13.9s
🟢 huge virtualized list reaches row 512 2m 1s
🟢 tests/flows.e2e.ts · 7 passed 55.7s
🟢 text inputs unlock the submit button 6.8s
🟢 filled text inputs report their values 6.0s
🟢 modal flow confirms through the native alert 9.8s
🟢 bottom tabs act inside the last tab 7.2s
🟢 error recovery retries, then confirms the delete 9.4s
🟢 choice controls place the exact order 7.1s
🟢 debounced search selects the target once results arrive 9.4s
🟢 tests/login-form.e2e.ts · 3 passed 31.3s
🟢 login form › rejects a malformed email 8.9s
🟢 login form › rejects an unknown account 8.8s
🟢 login form › signs in and logs out 13.6s
🟢 tests/smoke.e2e.ts · 1 passed 7.2s
🟢 home lists the scenarios and opens one 7.2s

Tested by e2e 0.23.0 · 11m 6s · android-emulator · run artifacts

@KrzysztofMoch

Copy link
Copy Markdown
Member Author

[claude-opus-5-5] responding on behalf of KrzysztofMoch

Two Cubic threads are left open for a human decision, both about the secrets invariant:

  • Symlinked artifacts: media: 'embed' and the code frames check paths lexically, not by canonical path. Only test, config, or engine code could plant a symlink under results/, and AGENTS.md gives that code full OS authority already, so I did not add a realpath check.
  • Multiline secrets in code frames: each line goes through redactFragments, which masks any 8+ character run of a secret. A line holding under 8 characters of a multiline secret can remain. Redacting the whole file instead would shift every later line number.

Also declined for a follow-up (in the PR body): folding --last-failed reruns and explore findings into the page, and streaming 'embed' past V8's string limit.

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets go

@okwasniewski
okwasniewski merged commit a133807 into main Nov 10, 2026
32 checks passed
@okwasniewski
okwasniewski deleted the krzysztof/html-reporter branch November 10, 2026 14:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:XXL 1,000+ changed lines, not counting tests, fixtures, translated docs, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants