Repository navigation
perf: decode and encode PNGs with @napi-rs/image - #1587
Conversation
e2e, @e2e-dev/mobile, and @e2e-dev/decision read and write PNGs with @napi-rs/image instead of fast-png and pngjs. Decoding is 8 to 15 times faster, encoding 5 to 8 times, and a toHaveScreenshot capture 4 to 12 times. - Chromium's 8-bit RGB captures are widened to RGBA by the addon, not by a JS loop. Other color types go through a short layout step. - e2e loads the addon when the first PNG is read or written, so --version, --help, and a run with no screenshot never need its native binding. A missing binding fails the call with IMAGE_CODEC_UNAVAILABLE. - pixelmatch stays: the addon has no diff API. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
🟢 e2e web: 91 passedAll 91 tests in 41 files
Tested by e2e 0.23.0 · 1m 1s · web · run artifacts |
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 87.6k tokens (93% cached) · $0.0038 All 33 tests in 5 files
Tested by e2e 0.23.0 · 2m 2s · web · run artifacts |
There was a problem hiding this comment.
2 issues found across 30 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".changeset/napi-rs-image.md">
<violation number="1" location=".changeset/napi-rs-image.md:2">
P2: These `patch` entries under-version a platform-breaking regression: on Linux ppc64, s390x, and riscv64, mobile and decision now fail during import, while e2e screenshot calls fail. The previous codecs were JavaScript packages; bump these affected 0.x packages to `minor` and start the summary with `**Breaking:**`.
(Based on your team's feedback about 0.x changeset bumps.) .</violation>
</file>
<file name="packages/decision/src/png.ts">
<violation number="1" location="packages/decision/src/png.ts:26">
P2: This pixel-layout conversion duplicates the corresponding helpers in `packages/mobile/src/png.ts` and `packages/e2e/src/internal/image.ts`. Move the shared conversion into a common internal helper so format and rounding fixes cannot drift between packages.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
| @@ -0,0 +1,68 @@ | |||
| import { endianness } from 'node:os'; | |||
There was a problem hiding this comment.
P2: This pixel-layout conversion duplicates the corresponding helpers in packages/mobile/src/png.ts and packages/e2e/src/internal/image.ts. Move the shared conversion into a common internal helper so format and rounding fixes cannot drift between packages.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/decision/src/png.ts, line 26:
<comment>This pixel-layout conversion duplicates the corresponding helpers in `packages/mobile/src/png.ts` and `packages/e2e/src/internal/image.ts`. Move the shared conversion into a common internal helper so format and rounding fixes cannot drift between packages.</comment>
<file context>
@@ -0,0 +1,68 @@
+ * widened by the addon: opacity 1 keeps every sample and adds the opaque
+ * alpha channel, several times faster than `spread` does it.
+ */
+export function decodeRgba(bytes: Uint8Array): Rgba {
+ const decoder = new Transformer(bytes);
+ const { width, height, format, colorType } = decoder.metadataSync();
</file context>
There was a problem hiding this comment.
[claude-sonnet-5-5] responding on behalf of kdy1
Left as three copies, and left open for the owner. @e2e-dev/mobile and @e2e-dev/decision reach e2e only through its public entries (AGENTS.md, Layout), so one shared helper means a new export on e2e or e2e/engine, or a new package. That is a public-contract and release decision, not part of a codec swap. readPngSize is already copied in web and mobile for the same reason. It is listed under the PR body decisions.
| "e2e": patch | ||
| "@e2e-dev/decision": patch | ||
| "@e2e-dev/mobile": patch | ||
| --- | ||
|
|
||
| PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque. |
There was a problem hiding this comment.
P2: These patch entries under-version a platform-breaking regression: on Linux ppc64, s390x, and riscv64, mobile and decision now fail during import, while e2e screenshot calls fail. The previous codecs were JavaScript packages; bump these affected 0.x packages to minor and start the summary with **Breaking:**.
(Based on your team's feedback about 0.x changeset bumps.) .
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/napi-rs-image.md, line 2:
<comment>These `patch` entries under-version a platform-breaking regression: on Linux ppc64, s390x, and riscv64, mobile and decision now fail during import, while e2e screenshot calls fail. The previous codecs were JavaScript packages; bump these affected 0.x packages to `minor` and start the summary with `**Breaking:**`.
(Based on your team's feedback about 0.x changeset bumps.) .</comment>
<file context>
@@ -0,0 +1,7 @@
+---
+"e2e": patch
+"@e2e-dev/decision": patch
+"@e2e-dev/mobile": patch
</file context>
| "e2e": patch | |
| "@e2e-dev/decision": patch | |
| "@e2e-dev/mobile": patch | |
| --- | |
| PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque. | |
| "e2e": minor | |
| "@e2e-dev/decision": minor | |
| "@e2e-dev/mobile": minor | |
| --- | |
| **Breaking:** PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque. |
There was a problem hiding this comment.
[claude-sonnet-5-5] responding on behalf of kdy1
Left open for the owner. AGENTS.md says never pick minor on our own, and whether losing Linux on ppc64, s390x, and riscv64 is a breaking change is a release call. It is the first item under "Decisions for the reviewer" in the PR body, and the changeset names the missing platforms.
… helpers - image.test.ts gains a 2-bit gray row and decision's overlay test a palette row, so every color type decodePng claims is exercised. - The six test helpers that decode with Transformer now throw unless the bytes are a PNG, as pngjs did, so a JPEG cannot pass for one. - The IMAGE_CODEC_UNAVAILABLE row in errors.mdx names the agent screenshots that need the addon: those the runner resizes, over 768 px. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 12 replayed from cache · 58 model calls · 368.0k tokens (82% cached) · $0.0247 All 22 tests in 1 file
Tested by e2e 0.23.0 · 8m 46s · ios-simulator · run artifacts |
🟢 e2e android agent: 1 flaky, 19 passed24 agent steps · 15 replayed from cache · 39 model calls · 285.7k tokens (84% cached) · $0.0187
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 19 passed | 24 steps · 39 calls | 4m 48s | |
| 🟢 | act completes Login Form | 1 step | 22.4s |
| 🟢 | act completes Infinite Scroll List | 1 step | 21.3s |
| 🟢 | act completes Modal Flow | 1 step | 9.8s |
| 🟢 | act completes Bottom Tabs | 1 step | 6.9s |
| 🟢 | act completes Text Input Variations | 1 step | 6.1s |
| 🟢 | act completes Broken Accessibility | 2 steps · 5 calls | 14.2s |
| 🟢 | act completes Vision Only | 2 steps · 1 call | 6.5s |
| act completes Gestures (1 failed attempt first) | 1 step · 9 calls | 22.4s | |
| 🟢 | act completes Async States | 1 step · 13 calls | 26.1s |
| 🟢 | act completes Debounced Search | 1 step | 6.7s |
| 🟢 | act completes Huge Virtualized List | 1 step | 1m 1s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 7.8s |
| 🟢 | act completes Flattened Login | 2 steps · 6 calls | 14.6s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 7.3s |
| 🟢 | act completes WebView Accessibility | 1 step · 4 calls | 13.4s |
| 🟢 | act completes Permission Prompt | 1 step | 6.3s |
| 🟢 | act completes Product Catalog | 1 step | 8.1s |
| 🟢 | act completes Error Recovery | 1 step | 8.8s |
| 🟢 | act completes Choice Controls | 1 step | 8.3s |
| 🟢 | act completes Sequential Onboarding | 1 step | 9.5s |
Tested by e2e 0.23.0 · 5m 18s · android-emulator · run artifacts
🟢 e2e android: 57 passed, 1 skippedAll 58 tests in 7 files
Tested by e2e 0.23.0 · 7m 39s · android-emulator · run artifacts |
🟢 e2e ios: 55 passed, 1 skippedAll 56 tests in 7 files
Tested by e2e 0.23.0 · 6m 25s · ios-simulator · run artifacts |
|
[claude-sonnet-5-5] responding on behalf of kdy1 Left for you:
|
okwasniewski
left a comment
There was a problem hiding this comment.
great! I think its fine to ship it as patch no need for a minor bump. good job!
e2e,@e2e-dev/mobile, and@e2e-dev/decisionread and write PNGs with@napi-rs/imageinstead offast-pngandpngjs(packages/e2e/src/internal/image.ts,packages/mobile/src/png.ts,packages/decision/src/png.ts).pixelmatchstays: the addon has no diff API.opacity(1)keeps every sample), which was most of the decode once the codec was native. Other color types go through a short layout step in JS.e2eloads the addon when the first PNG is read or written (createRequire, asoxc()does), so--version,--help, and a run that takes no screenshot never need the binding. A missing binding fails the call with the newIMAGE_CODEC_UNAVAILABLE(docs/reference/errors.mdx). TheSTARTUPtable inpackage-imports.test.tspins the lazy load.@e2e-dev/mobileand@e2e-dev/decisionimport the addon with the package. They are opt-in, and a lazy load there would turn a missing binding into a secure screenshot silently withheld (redactSecure) or a zoom silently skipped (decodeinoverlay.ts). Stopping at import is the louder failure.Speed
Main (
0c1b3572) against this branch: bothdistbuilds in one process, A/B then B/A, 11 rounds per cell, medians over three real screenshots per profile. Each case first checks that both sides produce the same pixels.downscalePixels(768)toHaveScreenshotcapture, whole screenmaskPng, 2 rectszoomAround(400, 2x)What is left at 2560×1440 is the JS box resample, unchanged here: 14.6 of the 28 ms.
pnpm bench:ab --suite testbed -- tests/visual.e2e.ts, 30 pairs. Every test ran the same steps with the same outcome, and every counter matched. The harness notes that the suite's tests differ between the builds becausescreenshot.e2e.tswas ported;visual.e2e.tsitself is unchanged.expect.toHaveScreenshote2e --version, 30 interleaved pairsDecisions for the reviewer
e2e, publishes 19. Linux on ppc64, s390x, and riscv64 (gnu and musl), armv7 musl, android arm32, and OpenHarmony have none. TheretoHaveScreenshotand any agent screenshot the runner has to resize (a side over 768 px) fail withIMAGE_CODEC_UNAVAILABLE, andmobileanddecisionfail at import. The changeset is apatch; whether this is a**Breaking:**minor is your call.FilterType.Adaptivemakes a Chromium capture about 15% larger thanfast-pngwrote it (55 to 63 KB at 1280×720) and a device screenshot about 20% smaller. "The smaller of no filter and adaptive" would never be larger thanfast-pngon a Chromium capture, costs about twice the encode time (5.3 to 9.7 ms at 1280×720), and touches the same three files. Not done here.LAYOUTS,narrow,spread) exists ine2e,mobile, anddecision.mobileanddecisionreache2eonly through its public entries, so sharing it means a new export one2eore2e/engine, or a new package.readPngSizeis copied in web and mobile for the same reason.fast-png,fflate, andiobuffer.pngjsstays throughpixelmatch.Transformerholds in a finalizer that Node runs a loop turn after V8 collects the owner, and it does not tell V8 about that memory. A loop that calls these functions back to back without yielding keeps about 70 MB per 2560×1440 capture: 120 calls peak at 8.4 GB where main stayed at 253 MB. Real runs yield between calls: worker peaks are 240 MB forvisual.e2e.ts(main 315 MB) and 490 MB forscreenshot.e2e.ts(main 427 MB).Verified
Ran it locally: yes
tests/visual.e2e.tsagainst the stored screenshotsfast-pngwrote: 4 passed, and no tracked file changed.@napi-rs/image-darwin-arm64unresolvable, which is what--omit=optionaldoes), on a testbed config withoutdecisionFilter:tests/todos.e2e.tspasses: 7 passed, exit 0.tests/visual.e2e.tsfails every test with exit 3:@e2e-dev/decision, so there the same missing binding stops at config load, exit 2:failed to load config .../e2e.config.ts: Cannot find native binding. npm has a bug related to optional dependencies (https://github.com/npm/cli/issues/4828). ...Gestures, which flaked the same way on perf: skip the MCP server, prompt library, and Playwright until a command needs them #1576.🤖 Generated with Claude Code