Repository navigation
feat(html-reporter): write the run as one HTML page - #1642
Conversation
🟢 e2e web: 91 passedAll 91 tests in 41 files
Tested by e2e 0.23.0 · 1m 1s · web · run artifacts |
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 88.3k tokens (94% cached) · $0.0035 All 33 tests in 5 files
Tested by e2e 0.23.0 · 1m 29s · web · run artifacts |
There was a problem hiding this comment.
2 issues found across 48 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/html-reporter/src/reporter.ts">
<violation number="1" location="packages/html-reporter/src/reporter.ts:112">
P1: The containment check is lexical, but `readFile` follows symlinks; an artifact inside `artifactsRoot` can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in `sourceFrames`.</violation>
<violation number="2" location="packages/html-reporter/src/reporter.ts:153">
P1: Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into `report.html`. Redact the complete frame text before splitting it into lines.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| const file = path.resolve(artifactsRoot, artifactPath); | ||
| const inside = path.relative(artifactsRoot, file); | ||
| if (inside === '' || inside.startsWith('..') || path.isAbsolute(inside)) continue; | ||
| const bytes = await readFile(file).catch((error: unknown) => { |
There was a problem hiding this comment.
P1: The containment check is lexical, but readFile follows symlinks; an artifact inside artifactsRoot can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in sourceFrames.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/html-reporter/src/reporter.ts, line 112:
<comment>The containment check is lexical, but `readFile` follows symlinks; an artifact inside `artifactsRoot` can point outside and be embedded in full. Check canonical paths against the canonical roots before reading, including in `sourceFrames`.</comment>
<file context>
@@ -0,0 +1,182 @@
+ const file = path.resolve(artifactsRoot, artifactPath);
+ const inside = path.relative(artifactsRoot, file);
+ if (inside === '' || inside.startsWith('..') || path.isAbsolute(inside)) continue;
+ const bytes = await readFile(file).catch((error: unknown) => {
+ if (error instanceof Error && 'code' in error && error.code === 'ENOENT') return undefined;
+ throw error;
</file context>
There was a problem hiding this comment.
[claude-opus-5-5] responding on behalf of KrzysztofMoch
Not changing this here; leaving it for the owner since it touches the secrets invariant. Every path under results/ is written by the runner, and the only way to plant a symlink there is test, config, or engine code, which AGENTS.md says already runs with the runner's full OS authority: it can read any file and write it into the report itself. The lexical check stops a report path from walking out of the results directory, which is the case a report can actually cause. sourceFrames reads lines of the project's own test files, then redacts them.
| frames[file] = Object.fromEntries( | ||
| [...lines].flatMap((line) => { | ||
| const content = all[line - 1]; | ||
| return content === undefined ? [] : [[String(line), redact(content).slice(0, MAX_LINE_CHARS)]]; |
There was a problem hiding this comment.
P1: Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into report.html. Redact the complete frame text before splitting it into lines.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/html-reporter/src/reporter.ts, line 153:
<comment>Redacting each line separately misses secrets that span line breaks, so a multiline value can be copied verbatim into `report.html`. Redact the complete frame text before splitting it into lines.</comment>
<file context>
@@ -0,0 +1,182 @@
+ frames[file] = Object.fromEntries(
+ [...lines].flatMap((line) => {
+ const content = all[line - 1];
+ return content === undefined ? [] : [[String(line), redact(content).slice(0, MAX_LINE_CHARS)]];
+ }),
+ );
</file context>
There was a problem hiding this comment.
[claude-opus-5-5] responding on behalf of KrzysztofMoch
Leaving this for the owner too. Each line goes through redactFragments, which also masks any run of 8 or more characters of a secret, so every line of a multiline value that carries 8+ characters of it is masked. Redacting the whole file first would replace a multiline value with one <secret:...> token and shift every later line number, so frames would point at the wrong code. What can remain is a line holding under 8 characters of a multiline secret.
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 42 model calls · 240.8k tokens (83% cached) · $0.0156 All 20 tests in 1 file
Tested by e2e 0.23.0 · 7m 20s · android-emulator · run artifacts |
🟢 e2e ios: 55 passed, 1 skippedAll 56 tests in 7 files
Tested by e2e 0.23.0 · 6m 53s · ios-simulator · run artifacts |
🟢 e2e ios agent: 1 flaky, 20 passed, 1 skipped25 agent steps · 12 replayed from cache · 59 model calls · 429.2k tokens (81% cached) · $0.0305
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 20 passed, 1 skipped | 25 steps · 59 calls | 9m 0s | |
| 🟢 | act completes Login Form | 1 step | 40.0s |
| 🟢 | act completes Infinite Scroll List | 1 step | 36.5s |
| 🟢 | act completes Modal Flow | 1 step | 10.0s |
| 🟢 | act completes Bottom Tabs | 1 step | 7.4s |
| 🟢 | act completes Text Input Variations | 1 step | 14.2s |
| 🟢 | act completes Broken Accessibility | 2 steps · 9 calls | 47.2s |
| 🟢 | act completes Vision Only | 2 steps · 6 calls | 31.5s |
| 🟢 | act completes Gestures | 1 step · 7 calls | 32.1s |
| 🟢 | act completes Async States | 1 step · 10 calls | 35.8s |
| 🟢 | act completes Debounced Search | 1 step · 1 call | 27.8s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 29.3s |
| 🟢 | act completes Flattened Login | 2 steps · 10 calls | 58.5s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 14.3s |
| 🟢 | act completes WebView Accessibility | 1 step | 12.7s |
| 🟢 | act completes Permission Prompt | 1 step | 11.7s |
| 🟢 | act completes Photo Picker | 1 step · 2 calls | 31.3s |
| 🟢 | act completes Product Catalog | 1 step | 10.1s |
| 🟢 | act completes Error Recovery | 1 step | 9.4s |
| 🟢 | act completes Choice Controls | 1 step · 5 calls | 36.6s |
| act completes Stripe PaymentSheet (1 failed attempt first) | 1 step · 8 calls | 32.1s | |
| 🟢 | act completes Apple Pay | 1 step | 11.8s |
| ⏭️ | act completes Apple Pay Billing Address (skipped: the CI simulator's Apple Pay sheet skips the billing-address step, so the scenario cannot be completed there) |
Tested by e2e 0.23.0 · 9m 35s · ios-simulator · run artifacts
There was a problem hiding this comment.
All reported issues were addressed across 12 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
🟢 e2e android: 57 passed, 1 skippedAll 58 tests in 7 files
Tested by e2e 0.23.0 · 11m 6s · android-emulator · run artifacts |
|
[claude-opus-5-5] responding on behalf of KrzysztofMoch Two Cubic threads are left open for a human decision, both about the secrets invariant:
Also declined for a follow-up (in the PR body): folding |
4f66707 to
ef79bc5
Compare
…mpts safely, and stop on the reporter signal
ef79bc5 to
878a691
Compare
e2e reports a run as terminal output, JSON, JUnit, markdown, or a pull request comment. None of them shows a test's steps next to its screenshots and video, and the Playwright and Maestro migration guides list
reporter: 'html'as missing.What changed
New package
@e2e-dev/html-reporter.htmlReporter()writes<output>/report.html, one page that opens from disk with no server:The page is a React app in
packages/html-reporter/app/. Vite andvite-plugin-singlefilebuild it into one HTML file with the fonts and icons inlined. The reporter (packages/html-reporter/src/reporter.ts) fills that template with the report-1 document, the media, and the test-file lines around each step.mediapicks how screenshots and videos reach the page:'link'(default, relative paths intoresults/),'embed'(one file to send), or'none'.The page shows the tests as a tree (file, describe blocks, tests, failures first) with status tabs and search. Each test shows its attempts, the error with expected and received values diffed, the code frame, its steps synced to the video, agent step details, and a copy prompt button that puts the failure on the clipboard for a coding agent.
Core:
FinishedRun.redact(text)(packages/e2e/src/types.ts). It masks configured secret values in text a reporter brings from outside the report, built from the same ledger asTestFilterContext.redact. The page uses it on the test-file lines it embeds. This needs the nexte2erelease, so the reporter'se2epeer floor is>=0.23.1.media: 'embed'leaves out downloads the runner could not scan for secrets (redaction: 'incomplete'), and download links save the file instead of opening it.Fonts are Geist and JetBrains Mono (OFL 1.1) and the icons are Lucide (ISC). The licence texts ship in the package and are listed in
NOTICE.Not covered yet (documented in
docs/reference/reporters.mdx): after--last-failedthe page shows only the rerun's tests, and undere2e exploreit shows no findings.Verified
Ran it locally: yes
Testbed, built CLI, with
htmlReporter()on the todos and forms suites plus one deliberately failing test, run with--repeat-each 2:The testbed's reporter stress suite (hostile titles, a 20 KiB message, timeouts, flakes, skips), plus a test that fills the configured
adminpassword and then fails, withmedia: 'embed'. Every one of the 16 test pages opened in Chromium with no page errors and no dialogs. The password value is nowhere inreport.html, and the frame showsfill('<secret:admin.password>').Pages from the web benchmark agent suite and the mobile benchmark on an iOS simulator, in both themes and at phone width.
Review
A fresh-context review found 9 issues. Fixed: unscanned downloads embedded in a page meant for sending,
--repeat-eachrows that looked identical, download links that opened in the report's origin, emoji splitting in the expected/received diff, and no runner-level test forrun.redact(packages/e2e/tests/integration/reporters.test.ts, which fails whenredactpasses text through).Declined:
--last-failedand explore findings into the page: documented above as a follow-up.'embed'past V8's string limit (about 400 MB of media): the docs already warn that device video makes an embedded page large.testscript builds first, andembedPayloadalready throws when the placeholder is not there exactly once.Cubic: fixed 15 threads (copied-prompt fence, cache-only step details, headline counted from the listed tests, the reporter abort signal, video part clock, serial member error fallback, a doubled screen-reader status, and docs wording). Two threads about the secrets invariant are left open for a human: symlinked artifacts, and multiline secrets in code frames.
Reviewed with sol-loop: 2 rounds with gpt-6.1-sol. Sol verdict: approved (before the fresh-context fixes above).
Closes ENG-1079.
Written with Claude Code (claude-opus-5-5).