Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .changeset/napi-rs-image.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
---
"e2e": patch
"@e2e-dev/decision": patch
"@e2e-dev/mobile": patch
---

PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque.
Comment on lines +2 to +7

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: These patch entries under-version a platform-breaking regression: on Linux ppc64, s390x, and riscv64, mobile and decision now fail during import, while e2e screenshot calls fail. The previous codecs were JavaScript packages; bump these affected 0.x packages to minor and start the summary with **Breaking:**.

(Based on your team's feedback about 0.x changeset bumps.) .

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At .changeset/napi-rs-image.md, line 2:

<comment>These `patch` entries under-version a platform-breaking regression: on Linux ppc64, s390x, and riscv64, mobile and decision now fail during import, while e2e screenshot calls fail. The previous codecs were JavaScript packages; bump these affected 0.x packages to `minor` and start the summary with `**Breaking:**`. 

(Based on your team's feedback about 0.x changeset bumps.) .</comment>

<file context>
@@ -0,0 +1,7 @@
+---
+"e2e": patch
+"@e2e-dev/decision": patch
+"@e2e-dev/mobile": patch
</file context>
Suggested change
"e2e": patch
"@e2e-dev/decision": patch
"@e2e-dev/mobile": patch
---
PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque.
"e2e": minor
"@e2e-dev/decision": minor
"@e2e-dev/mobile": minor
---
**Breaking:** PNG decoding and encoding now run on `@napi-rs/image`, a native addon with a prebuilt binary per platform, in place of `fast-png` and `pngjs`: decoding is 8 to 15 times faster, encoding 5 to 8 times, and a `toHaveScreenshot` capture 4 to 12 times. Stored screenshots decode to the same pixels as before. A screenshot the runner writes (a new baseline, or one rewritten because the screen changed) has the same pixels in different bytes: a Chromium capture comes out about 15% larger and a device screenshot about 20% smaller. The binary adds about 15 to 21 MB to an install, depending on the platform, and the package publishes none for Linux on ppc64, s390x, or riscv64. `e2e` loads it when the first screenshot is read or written, so `--version` and a run that takes none never need it, and a missing binding fails that call with `IMAGE_CODEC_UNAVAILABLE`; `@e2e-dev/mobile` and `@e2e-dev/decision` load it when they are imported. Gray PNGs with 1, 2, or 4 bits per sample now decode, and a 16-bit gray or RGB PNG with a transparent color now compares pixels to it at 16 bits, so a pixel that only rounds to it at 8 bits stays opaque.

@kdy1 kdy1 Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-sonnet-5-5] responding on behalf of kdy1

Left open for the owner. AGENTS.md says never pick minor on our own, and whether losing Linux on ppc64, s390x, and riscv64 is a breaking change is a release call. It is the first item under "Decisions for the reviewer" in the PR body, and the changeset names the missing platforms.

3 changes: 1 addition & 2 deletions apps/testbed/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,9 @@
},
"devDependencies": {
"@ai-sdk/devtools": "1.0.66",
"@napi-rs/image": "1.16.0",
"@types/node": "26.6.6",
"@types/pngjs": "6.0.5",
"ai": "7.0.163",
"pngjs": "7.0.0",
"typescript": "7.0.2",
"zod": "4.6.7"
}
Expand Down
36 changes: 27 additions & 9 deletions apps/testbed/tests/screenshot.e2e.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
* Adapted from microsoft/playwright tests/page/page-screenshot.spec.ts at v1.64.1.
* Copyright Microsoft Corporation. Licensed under the Apache License 2.0 (see ../app/playwright/LICENSE).
* Rewritten against the e2e public API: `page.screenshot()` is `app.screenshot()`, read back with
* pngjs, and a `mask` is the `toHaveScreenshot` option, compared against the same viewport with
* @napi-rs/image, and a `mask` is the `toHaveScreenshot` option, compared against the same viewport with
* boxes of the mask color painted over the masked elements. Playwright's golden images belong to
* its own renderer setup, so a golden comparison becomes a check of the pixels it shows.
*/
Expand Down Expand Up @@ -51,7 +51,7 @@ import { copyFileSync, mkdirSync, readFileSync, rmSync } from 'node:fs';
import path from 'node:path';
import { test, type Browser, type WebApp } from '@e2e-dev/web';
import { expect } from 'e2e';
import { PNG } from 'pngjs';
import { JsColorType, Transformer } from '@napi-rs/image';
import { content, CROSS, PREFIX } from './helpers.ts';

const stored = new Set<string>();
Expand All @@ -63,13 +63,31 @@ interface MaskTarget {
readonly index: number;
}

/** The PNG at `file`, decoded. */
function readPng(file: string): PNG {
return PNG.sync.read(readFileSync(file));
/** A decoded screenshot: 8-bit RGBA, rows top to bottom. */
interface Rgba {
readonly width: number;
readonly height: number;
readonly data: Uint8Array;
}

/** The RGB or RGBA PNG at `file`, decoded to RGBA. */
function readPng(file: string): Rgba {
const image = new Transformer(readFileSync(file));
const { width, height, format, colorType } = image.metadataSync();
if (format !== 'png') throw new Error(`${file} is not a PNG: the bytes are ${format}`);
const raw = image.rawPixelsSync();
if (colorType === JsColorType.Rgba8) return { width, height, data: raw };
if (colorType !== JsColorType.Rgb8) throw new Error(`${file} is neither an RGB nor an RGBA PNG: color type ${colorType}`);
const data = new Uint8Array(width * height * 4);
for (let from = 0, to = 0; from < raw.length; from += 3, to += 4) {
data.set(raw.subarray(from, from + 3), to);
data[to + 3] = 255;
}
return { width, height, data };
}

/** The red, green, and blue of the pixel at (`x`, `y`). */
function rgb(png: PNG, x: number, y: number): [number, number, number] {
function rgb(png: Rgba, x: number, y: number): [number, number, number] {
const offset = (y * png.width + x) * 4;
return [png.data[offset]!, png.data[offset + 1]!, png.data[offset + 2]!];
}
Expand All @@ -80,12 +98,12 @@ function near(actual: readonly number[], expected: readonly number[], slack = 3)
}

/** The pixels of `box` cut out of `png`, rounded out to whole pixels. */
function crop(png: PNG, box: { x: number; y: number; width: number; height: number }): Buffer {
function crop(png: Rgba, box: { x: number; y: number; width: number; height: number }): Buffer {
const left = Math.floor(box.x);
const top = Math.floor(box.y);
const right = Math.ceil(box.x + box.width);
const bottom = Math.ceil(box.y + box.height);
const rows: Buffer[] = [];
const rows: Uint8Array[] = [];
for (let y = top; y < bottom; y++) rows.push(png.data.subarray((y * png.width + left) * 4, (y * png.width + right) * 4));
return Buffer.concat(rows);
}
Expand Down Expand Up @@ -256,7 +274,7 @@ test.describe('page screenshot', { requires: ['browser'], tags: ['playwright'] }
}

/** Whether the line of `step` shows in `screenshot`: the canvas sits at the body margin, 8 pixels in. */
function hasLine(screenshot: PNG, step: number): boolean {
function hasLine(screenshot: Rgba, step: number): boolean {
const y = 8 + step * 100;
return [y - 1, y].some((row) => row >= 8 && rgb(screenshot, 8 + 150, row)[0] < 200);
}
Expand Down
1 change: 1 addition & 0 deletions docs/reference/errors.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,7 @@ These appear in terminal output and in the report as plain string codes.
| `WORKER_ENTRY_MISSING` | The worker entry module could not be resolved. Packaging fault. | infrastructure | 3 |
| `NODE_UNSUPPORTED` | The runtime cannot run e2e's TypeScript loader: a Node.js outside `^22.22.3 \|\| >=24.8.0`, Bun (no `module.registerHooks`), or Deno. The CLI checks this before it starts and exits 2 with the same sentence. The loader checks again where it registers its hooks, for a host that runs e2e's runner without the CLI | infrastructure | 3 |
| `TYPESCRIPT_COMPILER_UNAVAILABLE` | oxc, which compiles e2e's TypeScript, could not be loaded; the message gives the cause. Most often its native bindings for this platform (`@oxc-transform/binding-<platform>-<arch>`, `@oxc-parser/binding-<platform>-<arch>`) are not installed. npm leaves them out with `--omit=optional`, or when `package-lock.json` was written on another platform (npm/cli#4828). Install again without `--omit=optional`, or delete `package-lock.json` and `node_modules` and run `npm install` | infrastructure | 3 |
| `IMAGE_CODEC_UNAVAILABLE` | `@napi-rs/image`, which reads and writes the PNGs of screenshots, could not be loaded; the message gives the cause. It is loaded when the first screenshot is read or written, so a run that takes none never needs it. Most often its native binding for this platform (`@napi-rs/image-<platform>-<arch>*`) is not installed. npm leaves it out with `--omit=optional`, or when `package-lock.json` was written on another platform (npm/cli#4828). Install again without `--omit=optional`, or delete `package-lock.json` and `node_modules` and run `npm install`. The package publishes no binding for Linux on ppc64, s390x, or riscv64, where `toHaveScreenshot` cannot work, and neither can an agent screenshot the runner has to resize (one with a side over 768 px) | infrastructure | 3 |
| `ENGINE_FAILURE` | Any engine fault with no more specific mapping | infrastructure | 3 |
| `WORKER_PROTOCOL` | A worker reported the wrong unit, an attempt overlapped a live attempt, `openHost` tried to acquire an engine instance another host owns, or `host.openSession` tried to reuse a closed host or overlap an opening, live, or closing session | infrastructure | 3 |
| `SESSION_KEY_INVALID` | A worker received a session key that is not 32 bytes. Packaging or IPC fault. | infrastructure | 3 |
Expand Down
3 changes: 1 addition & 2 deletions packages/decision/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
"test": "vitest run"
},
"dependencies": {
"pngjs": "7.0.0",
"@napi-rs/image": "1.16.0",
"zod": "4.6.7"
},
"peerDependencies": {
Expand All @@ -43,7 +43,6 @@
"@ai-sdk/typesafe-ai": "3.0.34",
"@e2e-dev/web": "workspace:*",
"@types/node": "26.6.6",
"@types/pngjs": "6.0.5",
"ai": "7.0.163",
"ajv": "8.20.0",
"ajv-formats": "3.0.1",
Expand Down
18 changes: 10 additions & 8 deletions packages/decision/src/overlay.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { PNG } from 'pngjs';
import type { ExecutorPixels } from 'e2e';
import { fnv1a } from './hash.ts';
import { decodeRgba, encodeRgba, type Rgba } from './png.ts';
/** Pixels as the decision model receives them: the runner's capture without its masking count. */
export type Screenshot = Omit<ExecutorPixels, 'maskedRegionCount'>;
/** A viewport point in CSS pixels. */
Expand Down Expand Up @@ -54,7 +54,8 @@ export interface Zoom {
/**
* The `box` by `box` CSS-pixel region around `center`, kept inside the
* screenshot and enlarged `factor` times (nearest neighbor), for a finer
* closer look at a drawn control. Undefined when the pixels cannot be decoded.
* closer look at a drawn control. Undefined when the pixels cannot be
* decoded or the region comes out under one pixel, which no PNG can hold.
*/
export function zoomAround(screenshot: Screenshot, center: Point, box: number, factor: number): Zoom | undefined {
const png = decode(screenshot);
Expand All @@ -68,7 +69,8 @@ export function zoomAround(screenshot: Screenshot, center: Point, box: number, f
y: Math.round(Math.max(0, Math.min(cssHeight - side, center.y - side / 2))),
};
const size = Math.round(side * factor);
const out = new PNG({ width: size, height: size });
if (!Number.isFinite(size) || size < 1) return undefined;
const out = { width: size, height: size, data: new Uint8Array(size * size * 4) };
for (let y = 0; y < size; y += 1) {
for (let x = 0; x < size; x += 1) {
const sx = Math.min(png.width - 1, Math.floor((origin.x + x / factor) * scale));
Expand All @@ -81,7 +83,7 @@ export function zoomAround(screenshot: Screenshot, center: Point, box: number, f
out.data[to + 3] = 255;
}
}
return { screenshot: { mediaType: 'image/png', data: new Uint8Array(PNG.sync.write(out)), width: size, height: size, scale: 1 }, origin, factor };
return { screenshot: { mediaType: 'image/png', data: encodeRgba(out), width: size, height: size, scale: 1 }, origin, factor };
}
/**
* A hash of the `box` by `box` CSS-pixel region around `center`: what a tap
Expand Down Expand Up @@ -120,17 +122,17 @@ export function withGrid(screenshot: Screenshot, grid: Grid): Screenshot {
}
}
}
return { ...screenshot, data: new Uint8Array(PNG.sync.write(png)) };
return { ...screenshot, data: encodeRgba(png) };
}
function decode(screenshot: Screenshot): PNG | undefined {
function decode(screenshot: Screenshot): Rgba | undefined {
try {
return PNG.sync.read(Buffer.from(screenshot.data));
return decodeRgba(screenshot.data);
} catch {
return undefined;
}
}
const GRID: readonly [number, number, number] = [255, 0, 255];
function paint(png: PNG, x: number, y: number, color: readonly [number, number, number]): void {
function paint(png: Rgba, x: number, y: number, color: readonly [number, number, number]): void {
if (x < 0 || y < 0 || x >= png.width || y >= png.height) return;
const offset = (y * png.width + x) * 4;
png.data[offset] = color[0];
Expand Down
68 changes: 68 additions & 0 deletions packages/decision/src/png.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
import { endianness } from 'node:os';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This pixel-layout conversion duplicates the corresponding helpers in packages/mobile/src/png.ts and packages/e2e/src/internal/image.ts. Move the shared conversion into a common internal helper so format and rounding fixes cannot drift between packages.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At packages/decision/src/png.ts, line 26:

<comment>This pixel-layout conversion duplicates the corresponding helpers in `packages/mobile/src/png.ts` and `packages/e2e/src/internal/image.ts`. Move the shared conversion into a common internal helper so format and rounding fixes cannot drift between packages.</comment>

<file context>
@@ -0,0 +1,68 @@
+ * widened by the addon: opacity 1 keeps every sample and adds the opaque
+ * alpha channel, several times faster than `spread` does it.
+ */
+export function decodeRgba(bytes: Uint8Array): Rgba {
+  const decoder = new Transformer(bytes);
+  const { width, height, format, colorType } = decoder.metadataSync();
</file context>

@kdy1 kdy1 Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[claude-sonnet-5-5] responding on behalf of kdy1

Left as three copies, and left open for the owner. @e2e-dev/mobile and @e2e-dev/decision reach e2e only through its public entries (AGENTS.md, Layout), so one shared helper means a new export on e2e or e2e/engine, or a new package. That is a public-contract and release decision, not part of a codec swap. readPngSize is already copied in web and mobile for the same reason. It is listed under the PR body decisions.

import { FilterType, JsColorType, Transformer } from '@napi-rs/image';
/** An 8-bit RGBA image, rows top to bottom. */
export interface Rgba {
readonly width: number;
readonly height: number;
readonly data: Uint8Array;
}
const LITTLE_ENDIAN = endianness() === 'LE';
/** Channels per pixel, and whether each is a 16-bit sample, for every color type a decode reads: 8-bit RGB is widened by the addon before that. */
const LAYOUTS: Readonly<Partial<Record<JsColorType, { readonly channels: number; readonly wide: boolean }>>> = {
[JsColorType.L8]: { channels: 1, wide: false },
[JsColorType.La8]: { channels: 2, wide: false },
[JsColorType.Rgba8]: { channels: 4, wide: false },
[JsColorType.L16]: { channels: 1, wide: true },
[JsColorType.La16]: { channels: 2, wide: true },
[JsColorType.Rgb16]: { channels: 3, wide: true },
[JsColorType.Rgba16]: { channels: 4, wide: true },
};
/**
* Decodes PNG bytes to 8-bit RGBA, whatever the color type, bit depth, or
* palette; throws on bytes that are not a PNG. An 8-bit RGB screenshot is
* widened by the addon: opacity 1 keeps every sample and adds the opaque
* alpha channel, several times faster than `spread` does it.
*/
export function decodeRgba(bytes: Uint8Array): Rgba {
const decoder = new Transformer(bytes);
const { width, height, format, colorType } = decoder.metadataSync();
if (format !== 'png') throw new Error(`not a PNG: the bytes are ${format}`);
const rgb = colorType === JsColorType.Rgb8;
const raw = (rgb ? decoder.opacity(1) : decoder).rawPixelsSync();
const layout = LAYOUTS[rgb ? JsColorType.Rgba8 : colorType];
if (layout === undefined) throw new Error(`unsupported PNG color type ${colorType}`);
const { channels, wide } = layout;
if (raw.length !== width * height * channels * (wide ? 2 : 1)) throw new Error('the decoded PNG pixels do not fill its size');
const samples = new Uint8Array(raw.buffer, raw.byteOffset, raw.byteLength);
if (channels === 4 && !wide) return { width, height, data: samples };
return { width, height, data: spread(wide ? narrow(samples) : samples, channels, width * height) };
}
/**
* Encodes an image as PNG bytes, each row adaptively filtered: the addon's
* default writes no filter, which makes screenshots markedly larger.
*/
export function encodeRgba(image: Rgba): Uint8Array {
const png = Transformer.fromRgbaPixels(image.data, image.width, image.height).pngSync({ filterType: FilterType.Adaptive });
return new Uint8Array(png.buffer, png.byteOffset, png.byteLength);
}
/** 16-bit samples in native byte order as 8-bit samples, rounded. */
function narrow(bytes: Uint8Array): Uint8Array {
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const samples = new Uint8Array(bytes.length / 2);
for (let at = 0; at < samples.length; at += 1) samples[at] = Math.round((view.getUint16(at * 2, LITTLE_ENDIAN) * 255) / 65535);
return samples;
}
/** 8-bit samples of `channels` per pixel (gray, gray and alpha, RGB) as RGBA; a gray or RGB image is opaque. */
function spread(samples: Uint8Array, channels: number, pixels: number): Uint8Array {
const out = new Uint8Array(pixels * 4);
const gray = channels <= 2;
const alpha = channels === 2 || channels === 4;
for (let pixel = 0, from = 0, to = 0; pixel < pixels; pixel += 1, from += channels, to += 4) {
const r = samples[from]!;
out[to] = r;
out[to + 1] = gray ? r : samples[from + 1]!;
out[to + 2] = gray ? r : samples[from + 2]!;
out[to + 3] = alpha ? samples[from + channels - 1]! : 255;
}
return out;
}
18 changes: 8 additions & 10 deletions packages/decision/tests/executor-vision.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { describe, expect, it } from 'vitest';
import { decisionExecutor } from '../src/index.ts';
import { PNG } from 'pngjs';
import { BUTTONS, context, invalidConfig, scriptedDecision, scriptedOutputs, whitePixels } from './helpers.ts';
import { pngOf, rgbaOf } from './png-helpers.ts';


describe('vision', () => {
Expand Down Expand Up @@ -121,7 +121,7 @@ describe('vision details', () => {
const fixture = context({ tree: BUTTONS, observation: { pixels: whitePixels() }, model: text.model });
await decisionExecutor({ model, textModel: text.model, vision: true }).runStep(fixture.ctx);
const sent = (index: number) => {
const png = PNG.sync.read(Buffer.from(requests[index]?.files[0]?.data as Uint8Array));
const png = rgbaOf(requests[index]?.files[0]?.data as Uint8Array);
return { png, at: (x: number, y: number) => [...png.data.subarray((y * png.width + x) * 4, (y * png.width + x) * 4 + 3)] };
};
const full = sent(1);
Expand Down Expand Up @@ -191,10 +191,9 @@ describe('vision details', () => {
it('replays a landed point for the same control on the same page, and relocates one that missed', async () => {
/** Two screenshots that differ only inside the tapped region. */
const frame = (dot: boolean) => {
const png = new PNG({ width: 800, height: 600 });
png.data.fill(255);
if (dot) for (let i = 0; i < 4; i += 1) png.data[(100 * 800 + 100) * 4 + i] = 0;
return { data: new Uint8Array(PNG.sync.write(png)), mediaType: 'image/png' as const, width: 800, height: 600, scale: 1, maskedRegionCount: 0 };
const rgba = new Uint8Array(800 * 600 * 4).fill(255);
if (dot) for (let i = 0; i < 4; i += 1) rgba[(100 * 800 + 100) * 4 + i] = 0;
return { data: pngOf(800, 600, rgba), mediaType: 'image/png' as const, width: 800, height: 600, scale: 1, maskedRegionCount: 0 };
};
const { model, requests } = scriptedDecision((id, keys, call) => {
if (id === 'operation') return { choice: call < 12 ? 'tap_at' : 'blocked' };
Expand Down Expand Up @@ -229,10 +228,9 @@ describe('vision details', () => {
let look = 0;
fixture.observe.mockImplementation(async () => {
look += 1;
const png = new PNG({ width: 800, height: 600 });
png.data.fill(255);
png.data[look] = 0;
return { revision: String(look), text: 'x', truncated: false, viewport: { width: 800, height: 600 }, tree: BUTTONS, pixels: { data: new Uint8Array(PNG.sync.write(png)), mediaType: 'image/png' as const, width: 800, height: 600, scale: 1, maskedRegionCount: 0 } };
const rgba = new Uint8Array(800 * 600 * 4).fill(255);
rgba[look] = 0;
return { revision: String(look), text: 'x', truncated: false, viewport: { width: 800, height: 600 }, tree: BUTTONS, pixels: { data: pngOf(800, 600, rgba), mediaType: 'image/png' as const, width: 800, height: 600, scale: 1, maskedRegionCount: 0 } };
});
const verdict = await decisionExecutor({ model, textModel: text.model, vision: true }).runStep(fixture.ctx);
expect(verdict).toMatchObject({ status: 'blocked', summary: 'The decision model cannot make progress.' });
Expand Down
6 changes: 2 additions & 4 deletions packages/decision/tests/helpers.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import type { LanguageModel } from 'ai';
import type { DecisionExecutorOptions } from '../src/index.ts';
import type { ExecutorActions, ExecutorModelCall, ExecutorNode, ExecutorObservation, ExecutorPixels, JsonValue, StepExecutorContext, StepTurn } from 'e2e';
import { expect, vi } from 'vitest';
import { PNG } from 'pngjs';
import { whitePng } from './png-helpers.ts';
/** One scripted answer: the choice, its distribution, and the reported confidence. */
export interface ScriptedAnswer {
readonly choice: string;
Expand Down Expand Up @@ -187,7 +187,5 @@ export const FIELD: ExecutorNode = { id: 'root', children: [
]};
/** A decodable white screenshot of the fixture viewport. */
export function whitePixels(): ExecutorPixels {
const png = new PNG({ width: 800, height: 600 });
png.data.fill(255);
return { data: new Uint8Array(PNG.sync.write(png)), mediaType: 'image/png', width: 800, height: 600, scale: 1, maskedRegionCount: 0 };
return { data: whitePng(800, 600), mediaType: 'image/png', width: 800, height: 600, scale: 1, maskedRegionCount: 0 };
}
Loading
Loading