diff --git a/.changeset/skill-test-structure.md b/.changeset/skill-test-structure.md new file mode 100644 index 0000000000..d33ec93983 --- /dev/null +++ b/.changeset/skill-test-structure.md @@ -0,0 +1,5 @@ +--- +"e2e": patch +--- + +The bundled e2e skill now writes tests with one top-level `describe` per feature, one behavior per test, and arrange, act, assert order with every assertion at the end, instead of a check after each `agent.act`. diff --git a/docs/cache.mdx b/docs/cache.mdx index 1c4689fb72..453249cc46 100644 --- a/docs/cache.mdx +++ b/docs/cache.mdx @@ -25,16 +25,19 @@ run summary counts the outcomes: ## When a recording is written -Follow each `act` with a check of its outcome. In `read-write` mode, the -runner saves the recording at the end of the attempt when a later -verification step passed. +End the test with a check of its outcome. In `read-write` mode, the runner +saves the recording of each `act` at the end of the attempt when a later +verification step passed. One check at the end confirms every `act` before +it. ```ts title="tests/cart.e2e.ts" import { expect, test } from 'e2e'; test('adds an item', async ({ app, agent, screen }) => { await app.open('/cart'); + await agent.act('add one item to the cart'); + await expect(screen.getByRole('status')).toHaveText('1 item'); }); ``` @@ -53,7 +56,7 @@ when the attempt ended: | --- | --- | | `saved` | Recorded for the next run. | | `kept` | The entry stays as it is. | -| `unconfirmed` | Not saved: no check passed after the step. Follow it with a locator assertion or `agent.assert`. | +| `unconfirmed` | Not saved: no check passed after the step. End the test with a locator or engine assertion or `agent.assert`, or wait for the state with `locator.waitFor()` or `agent.waitFor`. | | `evicted` | The entry was deleted: its replay ended in a failure or a repair, or no check confirmed it before the attempt failed. | | `no-change` | Nothing recorded: no control appeared, went away, or changed state, and the route stayed the same. That includes a step whose only change is inside a [frame from another site](/browser#reach-into-an-iframe). Such a step runs live on every run. | | `not-written` | Not saved: the store rejected the write, or refused the recording, for example one over the file store's size limit. | @@ -61,7 +64,9 @@ when the attempt ended: A step that took no action records nothing and reports no `write`. A failing attempt still saves the steps verified before the failure. It -deletes unverified entries and the replays involved in the failure. A custom store needs a `delete` method for this. Cancelling +deletes unverified entries and the replays involved in the failure. To keep +setup goals recorded while you fix a failing assertion, end the setup with +a wait for its state, such as `locator.waitFor()`. A custom store needs a `delete` method for this. Cancelling a run leaves the cache unchanged. diff --git a/docs/core-concepts.mdx b/docs/core-concepts.mdx index cf6b2c2334..a3ef5bfa58 100644 --- a/docs/core-concepts.mdx +++ b/docs/core-concepts.mdx @@ -16,22 +16,26 @@ A test mixes three types of step: With a [custom executor](/executors), `act` and `assert` need no model. ```ts title="tests/todos.e2e.ts" -import { test, expect } from 'e2e'; +import { describe, test, expect } from 'e2e'; -test('todos survive a filter round-trip', async ({ app, agent, screen }) => { - await app.open('/todos'); +describe('todos', () => { + test('the Done filter shows only completed todos', async ({ app, agent, screen }) => { + await app.open('/todos'); + await agent.act('add todos "Buy milk" and "Walk the dog"'); + await agent.act('complete "Buy milk"'); + await agent.waitFor('"Buy milk" is completed and "Walk the dog" is open'); - await agent.act('add todos "Buy milk" and "Walk the dog"'); - await expect(screen.getByTestId('todo')).toHaveCount(2); + await screen.getByRole('tab', 'Done').tap(); - await agent.act('complete "Buy milk"'); - await agent.assert('only one todo remains open'); - - await screen.getByRole('tab', 'Done').tap(); - await expect(screen.getByTestId('todo')).toHaveCount(1); + await expect(screen.getByTestId('todo')).toHaveCount(1); + await agent.assert('the only todo listed is "Buy milk"'); + }); }); ``` +Each test checks one behavior in three blocks: arrange the starting state, +act on the behavior, then assert the result. + Await each agent call, screen call, and `expect.poll`. If a test returns while a step still runs, the test fails with `STEP_NOT_AWAITED`. @@ -55,26 +59,30 @@ await agent.act('open the Pro plan, set the seats to {seats}, and check out', { }); ``` -```ts title="Good: one atomic goal per call, each checked before the next" -await agent.act('open the Pro plan'); -await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible(); +```ts title="Good: one atomic goal per call, arrange, act, assert" +test('checks out the Pro plan', async ({ app, agent, screen }) => { + await app.open('/plans'); + await agent.act('open the Pro plan'); + await agent.act('set the seats to {seats}', { params: { seats: 3 } }); + await agent.waitFor('the cart shows the Pro plan with 3 seats'); -await agent.act('set the seats to {seats}', { params: { seats: 3 } }); -await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats'); + await agent.act('check out'); -await agent.act('check out'); -await expect(screen.getByText('Order confirmed')).toBeVisible(); + await expect(screen.getByText('Order confirmed')).toBeVisible(); +}); ``` -A failure then names the goal that broke. Each goal replays from its own +The setup goals come first and end with a wait for the state they reach. +The goal under test comes next. The assertions on its result come last. A +failure then names the goal that broke. Each goal replays from its own recording and gets its own model-call and action [budgets](/reference/agent#budgets). The actions toward one outcome are not goals: opening the screen or menu that holds a control, scrolling to it, closing a banner, or filling the fields of one form. - You control the order of the goals. The agent picks the actions. -- Check each goal's result with an `expect`. The [replay cache](/cache) - records a goal only after a later check passes. +- End the test with an `expect` on the result. The [replay cache](/cache) + records a goal only after a later check or wait passes. - Use the words that the screen shows. - Give test data in `params`. For a password, give a `Secret`. The model never gets its value as text. See [Authentication](/authentication). diff --git a/docs/self-healing.mdx b/docs/self-healing.mdx index 7851a14139..8d52ccc82d 100644 --- a/docs/self-healing.mdx +++ b/docs/self-healing.mdx @@ -101,7 +101,7 @@ clicks. If the discount button moved, the agent finds it. If discounts stop working, the step or the check after it fails. A goal can still pass in a way you didn't intend, for example with a -different discount. Follow each `act` with a deterministic check of the +different discount. End the test with a deterministic check of the result, such as the `toHaveText('10% off')` above. That check runs on every run, replayed or healed. diff --git a/plugins/e2e/skills/e2e/SKILL.md b/plugins/e2e/skills/e2e/SKILL.md index d82a51aeb4..4cbf31da09 100644 --- a/plugins/e2e/skills/e2e/SKILL.md +++ b/plugins/e2e/skills/e2e/SKILL.md @@ -81,14 +81,18 @@ export default { ```ts // tests/billing.e2e.ts -import { test } from '@e2e-dev/web'; +import { describe, test } from '@e2e-dev/web'; import { expect } from 'e2e'; -test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { - await app.open('/settings/billing'); - await agent.act('upgrade the workspace to the Pro plan'); - await expect(screen.getByRole('status')).toContainText('Pro'); - await expect(browser).toHaveURL('/settings/billing'); +describe('billing', () => { + test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { + await app.open('/settings/billing'); + + await agent.act('upgrade the workspace to the Pro plan'); + + await expect(screen.getByRole('status')).toContainText('Pro'); + await expect(browser).toHaveURL('/settings/billing'); + }); }); ``` @@ -106,13 +110,19 @@ test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { path's actions stay out of the test. A `screen` line is the `Test line:` its action printed, copied as is. When the outcome shows, run its `expect` with `run_code` and add it once it passes; a passing statement - also joins `test_lines`. The walk ends with the test written. Without a + also joins `test_lines`; checks go in the test's assert block at the end. + The walk ends with the test written. Without a server, learn the accessible names the app renders from the components or a `--headed` run. -3. A step is an `agent.act` goal or `screen` lines. Drive the flow with - `agent.act`, one atomic goal per call, and pin each outcome right after - with `expect` or `agent.assert`. A test with three goals makes three - calls. Start the test on the screen it covers, since every rerun pays +3. Shape the file before the steps (topic `writing-tests`, "Structure"): + one top-level `describe` per feature, and in it one test per behavior, + titled with the outcome it verifies. Each test runs arrange, act, assert, + in that order: reach the starting state, do the one thing under test, + then every `expect` or `agent.assert` at the end. A second outcome worth + checking is a second test, so a failing title says what broke. + A step is an `agent.act` goal or `screen` lines. Drive the flow with + `agent.act`, one atomic goal per call; three goals make three calls. + Start the test on the screen it covers, since every rerun pays for the steps before: in a browser a saved session instead of the sign-in form and `app.open` on that screen's path, on a device `device.openLink`, and data from the app's API (topic `writing-tests`). @@ -181,7 +191,8 @@ lines you changed: call's own `{ timeout }` still wins. An `expect` already gives up after 5 s by default (`assertionTimeout`). - Keep the replay cache on. A failed run keeps the recording of every goal - whose check passed, and the next run replays those with no model call. + a later check or wait confirmed (the wait closing the arrange block, topic + `writing-tests`), and the next run replays those with no model call. New, changed, and failed goals run live, and so does a replay the trace page shows as handed off. `agent.assert`, `agent.waitFor`, and `agent.extract` always call the model. Leave `--no-cache` off: it sends every goal to the model @@ -237,14 +248,23 @@ lines you changed: `credentials.user(name).password` or `secrets.get(name)` (separate namespaces: `secrets.get` never returns a password), and hand the opaque `Secret` only to `fill()` or `agent.act` params. +- Tests: one top-level `describe` per feature, one behavior per test, + arrange, act, assert with every assertion last (topic `writing-tests`). + Bad: one test that adds a todo, checks it, marks it done, and checks + again. Good: `'adds a todo to the list'` and `'marks a todo as done'`, + the second adding its todo in its arrange block. - Agent instructions: one goal per `act`, the wording on screen, real values in params. A goal is atomic: one outcome that one check confirms. Two - outcomes worth checking are two goals, so split them into two `act` calls - (topic `agent`). Bad: `'add a todo and mark it done'`. Good: `'add a - todo'`, check, then `'mark the todo as done'`, check. Judge meaning, not - phrasing: `toContain('Pro')`, not an exact sentence a model produced. -- Check each agent goal's outcome. A passing `act` with a recorded check can - be cached and replayed without model calls (topic `agent`). + outcomes are two goals in two `act` calls (topic `agent`), never `'add a + todo and mark it done'`. Judge meaning, not phrasing: `toContain('Pro')`, + not an exact sentence a model produced. +- End every test that runs an agent goal with a verifying check: a + locator matcher (`expect(locator)`), an engine matcher + (`expect(browser).toHaveURL`), or `agent.assert`. A passing one confirms + every `act` before it, so they can be cached and replayed without model + calls, and so do `locator.waitFor`, `browser.waitForURL`, and + `agent.waitFor`. Reads, `expect.poll`, and plain-value `expect` confirm + nothing (topic `agent`, "The replay cache"). - Shape the agent for this app: `context` for vocabulary the screens use, `system` for how it works, tools for a test API, named personas under `agents`. When a step fails, tighten the goal first, then the context, then diff --git a/plugins/e2e/skills/e2e/references/agent.md b/plugins/e2e/skills/e2e/references/agent.md index 9917b0a68b..62597ba249 100644 --- a/plugins/e2e/skills/e2e/references/agent.md +++ b/plugins/e2e/skills/e2e/references/agent.md @@ -95,12 +95,9 @@ context, so each persona records and replays its own steps. ## act: one goal ```ts -import { credentials, expect } from 'e2e'; +import { credentials } from 'e2e'; await agent.act('add a todo named "Buy milk"'); -await expect(screen.getByRole('listitem', 'Buy milk')).toBeVisible(); -await agent.act('mark the "Buy milk" todo as done'); -await expect(screen.getByRole('checkbox', 'Buy milk')).toBeChecked(); await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); const member = credentials.user('member'); @@ -295,7 +292,7 @@ elsewhere on the current screen. - One goal per `act`; goal order is the test's, the path the model's. - Keep each goal atomic: one outcome that one check confirms. When two parts of an instruction each leave a state worth checking, they are two - goals: give each its own `act` and check. Steps toward a single outcome + goals, each its own `act`. Steps toward a single outcome are the path, not more goals: scrolling to a control, closing a banner, opening the menu that holds it, filling the fields of one form. - A split flow fails at the goal that broke. Each goal replays from its own @@ -303,25 +300,38 @@ elsewhere on the current screen. timeout still caps them all. ```ts -// BAD: three goals in one instruction, one recording, no check between them. +// BAD: three goals in one instruction, one recording. await agent.act('open the Pro plan, set the seats to {seats}, and check out', { params: { seats: 3 } }); -// GOOD: three atomic goals, each pinned before the next. -await agent.act('open the Pro plan'); -await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible(); -await agent.act('set the seats to {seats}', { params: { seats: 3 } }); -await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats'); -await agent.act('check out'); -await expect(screen.getByText('Order confirmed')).toBeVisible(); +// GOOD: arrange with two atomic goals, act with the one under test, assert last. +test('checks out the Pro plan', async ({ app, agent, screen }) => { + await app.open('/plans'); + await agent.act('open the Pro plan'); + await agent.act('set the seats to {seats}', { params: { seats: 3 } }); + await agent.waitFor('the cart shows the Pro plan with 3 seats'); + + await agent.act('check out'); + + await expect(screen.getByText('Order confirmed')).toBeVisible(); + await expect(screen.getByText('Pro plan, 3 seats')).toBeVisible(); +}); ``` +Choosing the seats is its own behavior, so it gets its own test that ends +after `set the seats` and asserts the seat count. Tests follow arrange, act, +assert, one behavior each (topic `writing-tests`, "Structure"). + - Use the words on screen: `'open the Billing tab'`, not `'upgrade'`. - Values go in `params`, never expanded: `act('rename to {name}', { params })`. - Do not describe mechanics the runner handles: waiting, scrolling, retries. -- Pin every `act` outcome right after it; that check lets the cache record: +- End the test with a check of the outcome. A passing locator or engine + matcher, `agent.assert`, or wait confirms every `act` before it, which + lets the cache record them; close an arrange block of `act` goals with a + wait for its state, so a failure later in the test keeps them recorded: ```ts await agent.act('create a workspace named "Atlas" on the Pro plan'); + await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan'); ``` diff --git a/plugins/e2e/skills/e2e/references/bug-bash.md b/plugins/e2e/skills/e2e/references/bug-bash.md index 9e260aa66b..5eeddba3d3 100644 --- a/plugins/e2e/skills/e2e/references/bug-bash.md +++ b/plugins/e2e/skills/e2e/references/bug-bash.md @@ -220,21 +220,30 @@ verify one area after another. ```ts // tests/bugbash/cart-line-quantity.e2e.ts - import { test } from '@e2e-dev/web'; + import { describe, test } from '@e2e-dev/web'; import { credentials } from 'e2e'; - test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => { - const shopper = credentials.user('bb-cart'); - await app.open('/'); - await agent.act('sign in with the given credentials', { - params: { username: shopper.username, password: shopper.password }, + describe('cart', { tags: ['bugbash'] }, () => { + test('a cart line totals its quantity times the unit price', async ({ app, agent }) => { + const shopper = credentials.user('bb-cart'); + await app.open('/'); + await agent.act('sign in with the given credentials', { + params: { username: shopper.username, password: shopper.password }, + }); + await agent.act('add "Dune" to the cart'); + await agent.waitFor('you are signed in and the cart holds "Dune"'); + + await agent.act('set the quantity of "Dune" in the cart to 2'); + + await agent.assert('the cart line for "Dune" totals twice its unit price'); }); - await agent.assert('the app shows you are signed in'); - await agent.act('add "Dune" to the cart'); - await agent.act('set the quantity of "Dune" in the cart to 2'); - await agent.assert('the cart line for "Dune" totals twice its unit price'); }); ``` + + Write it in the shape topic `writing-tests` describes: arrange, closed by + a wait for the starting state, then the one action that shows the defect, + then the assertions. The wait keeps a broken setup from passing as the + defect, and keeps the setup goals cached across the failing runs. 3. Run your repros with the bug-bash config and an output directory of your own, `npx e2e run tests/bugbash/.e2e.ts --config e2e.bugbash.config.ts --output .e2e/bugbash/verify-` (a run clears @@ -244,7 +253,7 @@ verify one area after another. failed `act` whose message names the defect. A failed `act` carries the code the model chose, so only its message and screenshot tell the defect apart from a failure short of it. A blocked `act`, a timeout, a setup - error, or a failure that names something else means the repro is wrong: + error, a failed arrange wait, or a failure that names something else means the repro is wrong: sharpen the step and rerun. A passing test means the bug did not reproduce: reject the finding, say so, and move the test out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a diff --git a/plugins/e2e/skills/e2e/references/debugging.md b/plugins/e2e/skills/e2e/references/debugging.md index 21d40ee272..ea9fd66491 100644 --- a/plugins/e2e/skills/e2e/references/debugging.md +++ b/plugins/e2e/skills/e2e/references/debugging.md @@ -118,7 +118,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `agent.assert`, a false judgment (explained in the report) | Compare with the report's actual text or the screenshot; `{ timeout }` on the matcher; rewrite the question | | `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels, inside a frame from another site); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | | `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | -| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | +| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `waitFor` the condition right before the action that failed; close overlays; check `actionTimeout` | | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | | `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) | | `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way. Also GitHub Copilot refusing the configured model to the account (`model_not_supported`, a plan or settings limit the listing cannot show): reported once at the first call, the run stops | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`); for Copilot, pick an id the plan serves | @@ -143,7 +143,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `--step --headed` | A person steps through the test one key per step | | `--workers 1 --retries 0` | Take parallelism and retries out of the picture | | `--action-timeout 5000` | While writing a test: a wrong locator fails in 5 s instead of 30 s (`--assertion-timeout` for `expect`). Keep it longer than the slowest screen takes to load | -| `--no-cache` | Rule out a stale `agent.act` replay; one run, not every rerun. A failed run already deletes the replay that ran into the failure, so rerun with the cache on first. When the failure repeats after a replay the trace page shows as kept and the `--no-cache` run passes, tighten the check after that goal: the stale replay then fails it and is deleted | +| `--no-cache` | Rule out a stale `agent.act` replay; one run, not every rerun. A failed run already deletes the replay that ran into the failure, so rerun with the cache on first. When the failure repeats after a replay the trace page shows as kept and the `--no-cache` run passes, tighten the next check or wait after that goal (the arrange block's closing wait, or the assertions): the stale replay then fails it and is deleted | | `--debug` | Each agent step's duration, model calls, cost, transcript | | `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | What the model saw and called | | `--video`, `--video=retain-on-failure` | Watch the failed attempt; `step.startedAt` minus the segment's `startedAt` is the step's offset into it | diff --git a/plugins/e2e/skills/e2e/references/mcp.md b/plugins/e2e/skills/e2e/references/mcp.md index 1c91446c1e..3eabc8a6d6 100644 --- a/plugins/e2e/skills/e2e/references/mcp.md +++ b/plugins/e2e/skills/e2e/references/mcp.md @@ -125,9 +125,11 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. 1. Read topic `writing-tests` before you write or edit any test (`e2e://guide/writing-tests`, or `npx e2e guide writing-tests`); the server's instructions and every `open_session` result say so too. Then, - before `open_session`, create `tests/.e2e.ts` with its - imports, the `test(...)`, and the call that opens the screen it covers, - such as `app.open('/settings/members')`. + before `open_session`, create `tests/.e2e.ts` (its "Structure" + section) with its imports, the feature's `describe(...)`, a `test(...)` + titled with the one behavior it verifies, and the call that opens the + screen it covers, such as `app.open('/settings/members')`. A setup test + (`test.setup`) goes top-level instead, outside any `describe`. Then `open_session`, which opens the app, and go to that screen: on the web, `call {tool: "navigate", args: {url: ""}}`; on a device, the shortest path from the launch screen. Actions that only @@ -154,8 +156,8 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. With several `screen` steps on one screen, take their actions, then add the lines they printed in one edit. -3. When the outcome shows, add its check before the next step: an - `agent.assert`, or an `expect` you ran first with +3. When the outcome shows, add its check to the test's assert block, after + every step: an `agent.assert`, or an `expect` you ran first with `call {tool: "run_code", args: {code: "await expect(screen.getByRole('status')).toHaveText('Saved');"}}`. Build its locator from `observe`, or try it with `call {tool: "locate", args: {...}}`. One match: use the printed @@ -184,20 +186,26 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. `Test line:` or ran with `run_code` has run once on the live app, and the first run checks the rest. If you missed a line, `test_lines` returns every line printed or passed so far, in order, the act goals' paths - included. One walk, in order: + included, for the whole session rather than one test. A walk that goes + on to a second behavior goes on into a second test in the same + `describe`, with its own assert block: `navigate` back to the path its + `app.open` names first, so its lines run from the state the test starts + in. One walk, in order: ```text - write tests/members.e2e.ts: imports, test(...), app.open('/settings/members') + write tests/members.e2e.ts: imports, describe('members'), test('invites a member as an editor'), app.open('/settings/members') call open_session, then navigate {url: "/settings/members"} its line is the app.open already written edit + await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); call tap, type, select, tap the goal's path; its test lines stay out call run_code {code: "await expect(screen.getByRole('status')).toContainText('Invite sent');"} passed - edit + that line - call type {target: "n40", value: "ada"} prints the test line below - Test line: await screen.getByRole("searchbox", "Search members").fill("ada"); + edit + that line, in the assert block + write + test('finds a member by name'), app.open('/settings/members') + call navigate {url: "/settings/members"} the new test's fresh start + call type {target: "n40", value: "grace"} prints the test line below + Test line: await screen.getByRole("searchbox", "Search members").fill("grace"); edit + that line, as printed call run_code {code: "await expect(screen.getByRole('listitem')).toHaveCount(1);"} passed - edit + that line + edit + that line, in the assert block ``` 4. Run only that test: `run_tests {files: ["tests/.e2e.ts:"]}`, or diff --git a/plugins/e2e/skills/e2e/references/writing-tests.md b/plugins/e2e/skills/e2e/references/writing-tests.md index 06f4b3a2fa..798255ea44 100644 --- a/plugins/e2e/skills/e2e/references/writing-tests.md +++ b/plugins/e2e/skills/e2e/references/writing-tests.md @@ -3,6 +3,7 @@ ## Contents - A complete file +- Structure: one describe, arrange-act-assert, one behavior - Registration - Fixtures - Locators @@ -24,12 +25,19 @@ describe('todos', { tags: ['todos'] }, () => { await app.open('/todos'); }); - test('adds and completes a todo', async ({ agent, screen, browser }) => { + test('adds a todo to the list', async ({ agent, screen }) => { await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await expect(screen.getByRole('listitem')).toHaveCount(1); await expect(screen.getByRole('status', 'Remaining')).toHaveText('1 remaining'); + }); + + test('marks a todo as done', async ({ agent, screen, browser }) => { + await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await screen.getByRole('checkbox', 'Write the release notes').waitFor(); await agent.act('mark the todo as done'); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining'); await expect(browser).toHaveURL('/todos'); }); @@ -37,17 +45,64 @@ describe('todos', { tags: ['todos'] }, () => { test('ignores an empty submission', async ({ screen }) => { // An exact interaction: the empty submit is the point of the test. await screen.getByRole('button', 'Add').tap(); + await expect(screen.getByRole('listitem')).toHaveCount(0); }); }); ``` -The agent does the flow; `expect` pins what must be true after each goal, -and that check lets the replay cache rerun the step later. `screen` actions -are for exact interactions and values. Files match the config `tests` glob, -default `tests/**/*.e2e.ts`. In a browser every test starts from a fresh -context with no page open, so it calls `app.open()` first; on a device a test -that skips `app.open()` starts where the previous test left the app. +The agent does the flow; `expect` pins what must be true at the end, and +that check lets the replay cache rerun every goal before it later. `screen` +actions are for exact interactions and values. Files match the config +`tests` glob, default `tests/**/*.e2e.ts`. In a browser every test starts +from a fresh context with no page open, so it calls `app.open()` first; on a +device a test that skips `app.open()` starts where the previous test left +the app. + +## Structure: one describe, arrange-act-assert, one behavior + +Write every test in this shape. When a test fails, its title alone should +say what broke, with no debugging needed, unless the arrange or act steps +themselves failed or the app changed under them. + +- **One top-level `describe` per unit or feature.** Name it after the + feature (`describe('todos')`, `describe('checkout')`) and put every test + for it inside, with the hooks those tests share. A file holds one feature. + Nest a `describe` only to share a hook or an option (a `session`, + `agentContext`, `tags`) with some of its tests. Setup tests are the + exception: `test.setup` must be top-level, outside any `describe`. +- **Arrange, act, assert, in that order**, with a blank line between the + three blocks: + - Arrange reaches the state the behavior starts from: `session`, + `app.open` on the screen's path, data from a fixture, and a shared + `beforeEach`. When the app offers no shorter way, an `agent.act` or + `screen` lines can arrange too, as in `marks a todo as done` above. + End such an arrange block with a wait for the state it reached + (`locator.waitFor()`, `agent.waitFor`). A setup goal that passed + without doing its job then fails there, not at the outcome check, and + the wait confirms the setup goals for the replay cache. + - Act does the one thing under test: one `agent.act` goal, or the + `screen` lines of one interaction. A wait for the state the action + triggers (`agent.waitFor`, `locator.waitFor`) closes the act block. + - Assert holds every `expect`, `expect.poll`, and `agent.assert`, and + nothing after them. No check sits between arrange and act steps. A + setup test's `session.save` is bookkeeping and comes after its check. +- **One behavior per test.** A test verifies one outcome, and its title + names it: `'marks a todo as done'`, not `'todos work'` or `'adds and + completes a todo'`. An "and" in a title usually means two tests. Several + assertions are fine when they all check that one outcome (the count and + the remaining label after an add). A second behavior is a second test. + When it starts from the first one's result, its arrange block repeats the + first one's act (or seeds the result through the API) rather than + chaining both into one test. + +For the replay cache (topic `agent`), a passing locator or engine matcher, +`locator.waitFor`, `agent.assert`, or `agent.waitFor` confirms every +`agent.act` before it; `expect.poll` and a plain-value `expect` confirm +nothing, so end with a locator check too. When the test fails, every goal +after the last passing confirmation runs live again on the next run: the +arrange block's closing wait keeps the setup goals cached while you fix the +act or the assertions. ## Registration @@ -356,15 +411,18 @@ test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, scree ```ts // tests/dashboard.e2e.ts -import { test, expect } from 'e2e'; +import { describe, test, expect } from 'e2e'; -test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen }) => { - await app.open('/dashboard'); - await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +describe('dashboard', { session: 'admin' }, () => { + test('opens directly when signed in', async ({ app, screen }) => { + await app.open('/dashboard'); + + await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); + }); }); ``` -- Setup tests are top-level; exactly one setup saves a given name and the +- Setup tests are top-level, outside any `describe`; exactly one setup saves a given name and the body saves every declared name once (`SESSION_CONTRACT` otherwise). Names match `[A-Za-z0-9_.-]{1,128}`. - A session holds cookies, local storage, and IndexedDB for one run, @@ -430,7 +488,7 @@ test as close to its screen as the app allows: `call`. ```ts -import { test } from '@e2e-dev/web'; +import { describe, test } from '@e2e-dev/web'; import { expect } from 'e2e'; import { z } from 'zod'; @@ -446,11 +504,15 @@ const projectTest = test.extend<{ project: { id: string } }>({ }, }); -projectTest('renames a project', { session: 'admin' }, async ({ app, screen, project }) => { - await app.open(`/projects/${project.id}/settings`); // straight to the screen under test - await screen.getByLabel('Name').fill('Atlas'); - await screen.getByRole('button', 'Save').tap(); - await expect(screen.getByRole('status')).toHaveText('Saved'); +describe('project settings', { session: 'admin' }, () => { + projectTest('renames a project', async ({ app, screen, project }) => { + await app.open(`/projects/${project.id}/settings`); // straight to the screen under test + + await screen.getByLabel('Name').fill('Atlas'); + await screen.getByRole('button', 'Save').tap(); + + await expect(screen.getByRole('status')).toHaveText('Saved'); + }); }); ``` @@ -460,7 +522,9 @@ in the app. Grow a test that spans several screens one screen at a time when you have not checked its steps live: write up to the next screen you have not seen -pass, run it, then add the next. A passing locator step takes milliseconds, +pass, end it with a temporary check of that screen, run it, then replace +the check with the next part. The finished test keeps only its outcome +checks, in the assert block. A passing locator step takes milliseconds, and an `agent.act` that a later check confirmed replays from the cache with no model call. The cache keeps those entries when a later line fails, so each run pays model calls only for its new part. That holds for a first diff --git a/plugins/e2e/skills/migrate-to-e2e/SKILL.md b/plugins/e2e/skills/migrate-to-e2e/SKILL.md index 5e35781ece..8e8938ef20 100644 --- a/plugins/e2e/skills/migrate-to-e2e/SKILL.md +++ b/plugins/e2e/skills/migrate-to-e2e/SKILL.md @@ -194,13 +194,33 @@ fixture (topic `writing-tests`). Record each in the progress file. Then each test, one at a time: -- Keep its title, and open the file with a comment naming the source +- Open the file with a comment naming the source (`// Ported from __e2e__/flows/login.yml`), so reports and reviewers map one to the other. - Translate literally first with the guide's tables, one awaited call per source step. Each source assertion maps to its own e2e assertion. A wait that fails on timeout (`waitForFunction`) becomes one more, so the e2e count is at least the inventory's. +- Then, before the first run, arrange the translated lines in the shape the + e2e skill's `writing-tests` topic gives ("Structure"). Lines move, split + tests copy the steps they need, and closing waits are added; no source + step or assertion is dropped: + - One top-level `describe` per source file or feature, named after it. + Setup tests stay top-level. + - A source test that checks one behavior keeps its title. A source test + that checks several (a check, more steps, another check) splits into + one test per behavior, each titled with its outcome; the file comment + names the source test each came from. Each split test repeats the + steps before its behavior in its own arrange block, or reaches that + state through a `beforeEach`, a fixture, or the app's API. + - Each test runs arrange, act, assert: the steps that reach the starting + state, the steps of the one behavior, then its assertions, last. + - A source assertion that only guarded a precondition (the dashboard + showed before the real steps began) becomes the wait that closes the + arrange block (`locator.waitFor()`, `agent.waitFor`). It still counts + toward the assertion count, since it fails on timeout. + - Splitting adds tests, never removes a check: the e2e count across the + split tests is still at least the source's. - A test the source skips is ported in full and stays skipped, with the source's reason, so it can be turned back on in either suite. Its row says so; when the reason allows, run it unskipped once and note the @@ -238,7 +258,9 @@ Then each test, one at a time: `agent.act` comes after the literal port passes, never instead of it, and only with a model configured. Propose it for a stretch the source kept patching: coordinate taps, picker spins, long tap sequences through a -wizard. Pin the outcome right after with a locator assertion (topic `agent`). +wizard. Keep the test's shape: an `agent.act` in the arrange block ends it +with a wait for the state it reached, and one under test is checked by the +locator assertions at the end (topic `agent`). ## 6. Prove each port diff --git a/skills/e2e/SKILL.md b/skills/e2e/SKILL.md index d82a51aeb4..4cbf31da09 100644 --- a/skills/e2e/SKILL.md +++ b/skills/e2e/SKILL.md @@ -81,14 +81,18 @@ export default { ```ts // tests/billing.e2e.ts -import { test } from '@e2e-dev/web'; +import { describe, test } from '@e2e-dev/web'; import { expect } from 'e2e'; -test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { - await app.open('/settings/billing'); - await agent.act('upgrade the workspace to the Pro plan'); - await expect(screen.getByRole('status')).toContainText('Pro'); - await expect(browser).toHaveURL('/settings/billing'); +describe('billing', () => { + test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { + await app.open('/settings/billing'); + + await agent.act('upgrade the workspace to the Pro plan'); + + await expect(screen.getByRole('status')).toContainText('Pro'); + await expect(browser).toHaveURL('/settings/billing'); + }); }); ``` @@ -106,13 +110,19 @@ test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => { path's actions stay out of the test. A `screen` line is the `Test line:` its action printed, copied as is. When the outcome shows, run its `expect` with `run_code` and add it once it passes; a passing statement - also joins `test_lines`. The walk ends with the test written. Without a + also joins `test_lines`; checks go in the test's assert block at the end. + The walk ends with the test written. Without a server, learn the accessible names the app renders from the components or a `--headed` run. -3. A step is an `agent.act` goal or `screen` lines. Drive the flow with - `agent.act`, one atomic goal per call, and pin each outcome right after - with `expect` or `agent.assert`. A test with three goals makes three - calls. Start the test on the screen it covers, since every rerun pays +3. Shape the file before the steps (topic `writing-tests`, "Structure"): + one top-level `describe` per feature, and in it one test per behavior, + titled with the outcome it verifies. Each test runs arrange, act, assert, + in that order: reach the starting state, do the one thing under test, + then every `expect` or `agent.assert` at the end. A second outcome worth + checking is a second test, so a failing title says what broke. + A step is an `agent.act` goal or `screen` lines. Drive the flow with + `agent.act`, one atomic goal per call; three goals make three calls. + Start the test on the screen it covers, since every rerun pays for the steps before: in a browser a saved session instead of the sign-in form and `app.open` on that screen's path, on a device `device.openLink`, and data from the app's API (topic `writing-tests`). @@ -181,7 +191,8 @@ lines you changed: call's own `{ timeout }` still wins. An `expect` already gives up after 5 s by default (`assertionTimeout`). - Keep the replay cache on. A failed run keeps the recording of every goal - whose check passed, and the next run replays those with no model call. + a later check or wait confirmed (the wait closing the arrange block, topic + `writing-tests`), and the next run replays those with no model call. New, changed, and failed goals run live, and so does a replay the trace page shows as handed off. `agent.assert`, `agent.waitFor`, and `agent.extract` always call the model. Leave `--no-cache` off: it sends every goal to the model @@ -237,14 +248,23 @@ lines you changed: `credentials.user(name).password` or `secrets.get(name)` (separate namespaces: `secrets.get` never returns a password), and hand the opaque `Secret` only to `fill()` or `agent.act` params. +- Tests: one top-level `describe` per feature, one behavior per test, + arrange, act, assert with every assertion last (topic `writing-tests`). + Bad: one test that adds a todo, checks it, marks it done, and checks + again. Good: `'adds a todo to the list'` and `'marks a todo as done'`, + the second adding its todo in its arrange block. - Agent instructions: one goal per `act`, the wording on screen, real values in params. A goal is atomic: one outcome that one check confirms. Two - outcomes worth checking are two goals, so split them into two `act` calls - (topic `agent`). Bad: `'add a todo and mark it done'`. Good: `'add a - todo'`, check, then `'mark the todo as done'`, check. Judge meaning, not - phrasing: `toContain('Pro')`, not an exact sentence a model produced. -- Check each agent goal's outcome. A passing `act` with a recorded check can - be cached and replayed without model calls (topic `agent`). + outcomes are two goals in two `act` calls (topic `agent`), never `'add a + todo and mark it done'`. Judge meaning, not phrasing: `toContain('Pro')`, + not an exact sentence a model produced. +- End every test that runs an agent goal with a verifying check: a + locator matcher (`expect(locator)`), an engine matcher + (`expect(browser).toHaveURL`), or `agent.assert`. A passing one confirms + every `act` before it, so they can be cached and replayed without model + calls, and so do `locator.waitFor`, `browser.waitForURL`, and + `agent.waitFor`. Reads, `expect.poll`, and plain-value `expect` confirm + nothing (topic `agent`, "The replay cache"). - Shape the agent for this app: `context` for vocabulary the screens use, `system` for how it works, tools for a test API, named personas under `agents`. When a step fails, tighten the goal first, then the context, then diff --git a/skills/e2e/references/agent.md b/skills/e2e/references/agent.md index 9917b0a68b..62597ba249 100644 --- a/skills/e2e/references/agent.md +++ b/skills/e2e/references/agent.md @@ -95,12 +95,9 @@ context, so each persona records and replays its own steps. ## act: one goal ```ts -import { credentials, expect } from 'e2e'; +import { credentials } from 'e2e'; await agent.act('add a todo named "Buy milk"'); -await expect(screen.getByRole('listitem', 'Buy milk')).toBeVisible(); -await agent.act('mark the "Buy milk" todo as done'); -await expect(screen.getByRole('checkbox', 'Buy milk')).toBeChecked(); await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); const member = credentials.user('member'); @@ -295,7 +292,7 @@ elsewhere on the current screen. - One goal per `act`; goal order is the test's, the path the model's. - Keep each goal atomic: one outcome that one check confirms. When two parts of an instruction each leave a state worth checking, they are two - goals: give each its own `act` and check. Steps toward a single outcome + goals, each its own `act`. Steps toward a single outcome are the path, not more goals: scrolling to a control, closing a banner, opening the menu that holds it, filling the fields of one form. - A split flow fails at the goal that broke. Each goal replays from its own @@ -303,25 +300,38 @@ elsewhere on the current screen. timeout still caps them all. ```ts -// BAD: three goals in one instruction, one recording, no check between them. +// BAD: three goals in one instruction, one recording. await agent.act('open the Pro plan, set the seats to {seats}, and check out', { params: { seats: 3 } }); -// GOOD: three atomic goals, each pinned before the next. -await agent.act('open the Pro plan'); -await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible(); -await agent.act('set the seats to {seats}', { params: { seats: 3 } }); -await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats'); -await agent.act('check out'); -await expect(screen.getByText('Order confirmed')).toBeVisible(); +// GOOD: arrange with two atomic goals, act with the one under test, assert last. +test('checks out the Pro plan', async ({ app, agent, screen }) => { + await app.open('/plans'); + await agent.act('open the Pro plan'); + await agent.act('set the seats to {seats}', { params: { seats: 3 } }); + await agent.waitFor('the cart shows the Pro plan with 3 seats'); + + await agent.act('check out'); + + await expect(screen.getByText('Order confirmed')).toBeVisible(); + await expect(screen.getByText('Pro plan, 3 seats')).toBeVisible(); +}); ``` +Choosing the seats is its own behavior, so it gets its own test that ends +after `set the seats` and asserts the seat count. Tests follow arrange, act, +assert, one behavior each (topic `writing-tests`, "Structure"). + - Use the words on screen: `'open the Billing tab'`, not `'upgrade'`. - Values go in `params`, never expanded: `act('rename to {name}', { params })`. - Do not describe mechanics the runner handles: waiting, scrolling, retries. -- Pin every `act` outcome right after it; that check lets the cache record: +- End the test with a check of the outcome. A passing locator or engine + matcher, `agent.assert`, or wait confirms every `act` before it, which + lets the cache record them; close an arrange block of `act` goals with a + wait for its state, so a failure later in the test keeps them recorded: ```ts await agent.act('create a workspace named "Atlas" on the Pro plan'); + await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan'); ``` diff --git a/skills/e2e/references/bug-bash.md b/skills/e2e/references/bug-bash.md index 9e260aa66b..5eeddba3d3 100644 --- a/skills/e2e/references/bug-bash.md +++ b/skills/e2e/references/bug-bash.md @@ -220,21 +220,30 @@ verify one area after another. ```ts // tests/bugbash/cart-line-quantity.e2e.ts - import { test } from '@e2e-dev/web'; + import { describe, test } from '@e2e-dev/web'; import { credentials } from 'e2e'; - test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => { - const shopper = credentials.user('bb-cart'); - await app.open('/'); - await agent.act('sign in with the given credentials', { - params: { username: shopper.username, password: shopper.password }, + describe('cart', { tags: ['bugbash'] }, () => { + test('a cart line totals its quantity times the unit price', async ({ app, agent }) => { + const shopper = credentials.user('bb-cart'); + await app.open('/'); + await agent.act('sign in with the given credentials', { + params: { username: shopper.username, password: shopper.password }, + }); + await agent.act('add "Dune" to the cart'); + await agent.waitFor('you are signed in and the cart holds "Dune"'); + + await agent.act('set the quantity of "Dune" in the cart to 2'); + + await agent.assert('the cart line for "Dune" totals twice its unit price'); }); - await agent.assert('the app shows you are signed in'); - await agent.act('add "Dune" to the cart'); - await agent.act('set the quantity of "Dune" in the cart to 2'); - await agent.assert('the cart line for "Dune" totals twice its unit price'); }); ``` + + Write it in the shape topic `writing-tests` describes: arrange, closed by + a wait for the starting state, then the one action that shows the defect, + then the assertions. The wait keeps a broken setup from passing as the + defect, and keeps the setup goals cached across the failing runs. 3. Run your repros with the bug-bash config and an output directory of your own, `npx e2e run tests/bugbash/.e2e.ts --config e2e.bugbash.config.ts --output .e2e/bugbash/verify-` (a run clears @@ -244,7 +253,7 @@ verify one area after another. failed `act` whose message names the defect. A failed `act` carries the code the model chose, so only its message and screenshot tell the defect apart from a failure short of it. A blocked `act`, a timeout, a setup - error, or a failure that names something else means the repro is wrong: + error, a failed arrange wait, or a failure that names something else means the repro is wrong: sharpen the step and rerun. A passing test means the bug did not reproduce: reject the finding, say so, and move the test out of `tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a diff --git a/skills/e2e/references/debugging.md b/skills/e2e/references/debugging.md index 21d40ee272..ea9fd66491 100644 --- a/skills/e2e/references/debugging.md +++ b/skills/e2e/references/debugging.md @@ -118,7 +118,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `ASSERTION_FAILED` | Wrong expectation, or the state settles later than 5 s; for `agent.assert`, a false judgment (explained in the report) | Compare with the report's actual text or the screenshot; `{ timeout }` on the matcher; rewrite the question | | `ASSERTION_INCONCLUSIVE` | `agent.assert` or `agent.extract` asked about what the screen does not show (another page, still loading, only in pixels, inside a frame from another site); a failure, never a pass | `app.open()` or `agent.waitFor` the right screen first; ask about what is shown; `vision: true` when the answer is in pixels (the message says so) | | `MODEL_OUTPUT_INVALID` | An `extract` or `assert` answer failed the schema after one repair round | Simplify the schema or question; pick a stronger model | -| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `expect` the condition first; close overlays; check `actionTimeout` | +| `ACTION_FAILED` | Element not actionable (covered, disabled, detached), or an operation timed out | `waitFor` the condition right before the action that failed; close overlays; check `actionTimeout` | | `TEST_TIMEOUT` | The attempt exceeded `timeout` (120 s) | Split the test, or raise `timeout` | | `STEP_NOT_AWAITED` | The body returned while a step or `expect.poll` still ran, or an `afterEach` hook or fixture teardown returned while an `expect.poll` still ran: a call without `await` | `await` the call the code frame names (every `app`, `agent`, `screen`, `expect`, `expect.poll` call) | | `MODEL_UNAVAILABLE` | `agents..model` holds no AI SDK instance: reported once under `run.errors` at the first `agent` fixture; the run stops. A model-less custom executor fails only `waitFor` and `extract` this way. Also GitHub Copilot refusing the configured model to the account (`model_not_supported`, a plan or settings limit the listing cannot show): reported once at the first call, the run stops | Construct one, e.g. `gateway('openai/gpt-6-luna-fast')` from `ai`, and export its key (`AI_GATEWAY_API_KEY`); for Copilot, pick an id the plan serves | @@ -143,7 +143,7 @@ jq '.run.results[] | select(.selected and .status != "passed") | .attempts[-1] | `--step --headed` | A person steps through the test one key per step | | `--workers 1 --retries 0` | Take parallelism and retries out of the picture | | `--action-timeout 5000` | While writing a test: a wrong locator fails in 5 s instead of 30 s (`--assertion-timeout` for `expect`). Keep it longer than the slowest screen takes to load | -| `--no-cache` | Rule out a stale `agent.act` replay; one run, not every rerun. A failed run already deletes the replay that ran into the failure, so rerun with the cache on first. When the failure repeats after a replay the trace page shows as kept and the `--no-cache` run passes, tighten the check after that goal: the stale replay then fails it and is deleted | +| `--no-cache` | Rule out a stale `agent.act` replay; one run, not every rerun. A failed run already deletes the replay that ran into the failure, so rerun with the cache on first. When the failure repeats after a replay the trace page shows as kept and the `--no-cache` run passes, tighten the next check or wait after that goal (the arrange block's closing wait, or the assertions): the stale replay then fails it and is deleted | | `--debug` | Each agent step's duration, model calls, cost, transcript | | `--ai-trace`, then `npx unbox-ai runs .e2e/ai-trace.json` | What the model saw and called | | `--video`, `--video=retain-on-failure` | Watch the failed attempt; `step.startedAt` minus the segment's `startedAt` is the step's offset into it | diff --git a/skills/e2e/references/mcp.md b/skills/e2e/references/mcp.md index 1c91446c1e..3eabc8a6d6 100644 --- a/skills/e2e/references/mcp.md +++ b/skills/e2e/references/mcp.md @@ -125,9 +125,11 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. 1. Read topic `writing-tests` before you write or edit any test (`e2e://guide/writing-tests`, or `npx e2e guide writing-tests`); the server's instructions and every `open_session` result say so too. Then, - before `open_session`, create `tests/.e2e.ts` with its - imports, the `test(...)`, and the call that opens the screen it covers, - such as `app.open('/settings/members')`. + before `open_session`, create `tests/.e2e.ts` (its "Structure" + section) with its imports, the feature's `describe(...)`, a `test(...)` + titled with the one behavior it verifies, and the call that opens the + screen it covers, such as `app.open('/settings/members')`. A setup test + (`test.setup`) goes top-level instead, outside any `describe`. Then `open_session`, which opens the app, and go to that screen: on the web, `call {tool: "navigate", args: {url: ""}}`; on a device, the shortest path from the launch screen. Actions that only @@ -154,8 +156,8 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. With several `screen` steps on one screen, take their actions, then add the lines they printed in one edit. -3. When the outcome shows, add its check before the next step: an - `agent.assert`, or an `expect` you ran first with +3. When the outcome shows, add its check to the test's assert block, after + every step: an `agent.assert`, or an `expect` you ran first with `call {tool: "run_code", args: {code: "await expect(screen.getByRole('status')).toHaveText('Saved');"}}`. Build its locator from `observe`, or try it with `call {tool: "locate", args: {...}}`. One match: use the printed @@ -184,20 +186,26 @@ Resources: `e2e://guide` and `e2e://guide/` hold this skill. `Test line:` or ran with `run_code` has run once on the live app, and the first run checks the rest. If you missed a line, `test_lines` returns every line printed or passed so far, in order, the act goals' paths - included. One walk, in order: + included, for the whole session rather than one test. A walk that goes + on to a second behavior goes on into a second test in the same + `describe`, with its own assert block: `navigate` back to the path its + `app.open` names first, so its lines run from the state the test starts + in. One walk, in order: ```text - write tests/members.e2e.ts: imports, test(...), app.open('/settings/members') + write tests/members.e2e.ts: imports, describe('members'), test('invites a member as an editor'), app.open('/settings/members') call open_session, then navigate {url: "/settings/members"} its line is the app.open already written edit + await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } }); call tap, type, select, tap the goal's path; its test lines stay out call run_code {code: "await expect(screen.getByRole('status')).toContainText('Invite sent');"} passed - edit + that line - call type {target: "n40", value: "ada"} prints the test line below - Test line: await screen.getByRole("searchbox", "Search members").fill("ada"); + edit + that line, in the assert block + write + test('finds a member by name'), app.open('/settings/members') + call navigate {url: "/settings/members"} the new test's fresh start + call type {target: "n40", value: "grace"} prints the test line below + Test line: await screen.getByRole("searchbox", "Search members").fill("grace"); edit + that line, as printed call run_code {code: "await expect(screen.getByRole('listitem')).toHaveCount(1);"} passed - edit + that line + edit + that line, in the assert block ``` 4. Run only that test: `run_tests {files: ["tests/.e2e.ts:"]}`, or diff --git a/skills/e2e/references/writing-tests.md b/skills/e2e/references/writing-tests.md index 06f4b3a2fa..798255ea44 100644 --- a/skills/e2e/references/writing-tests.md +++ b/skills/e2e/references/writing-tests.md @@ -3,6 +3,7 @@ ## Contents - A complete file +- Structure: one describe, arrange-act-assert, one behavior - Registration - Fixtures - Locators @@ -24,12 +25,19 @@ describe('todos', { tags: ['todos'] }, () => { await app.open('/todos'); }); - test('adds and completes a todo', async ({ agent, screen, browser }) => { + test('adds a todo to the list', async ({ agent, screen }) => { await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await expect(screen.getByRole('listitem')).toHaveCount(1); await expect(screen.getByRole('status', 'Remaining')).toHaveText('1 remaining'); + }); + + test('marks a todo as done', async ({ agent, screen, browser }) => { + await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } }); + await screen.getByRole('checkbox', 'Write the release notes').waitFor(); await agent.act('mark the todo as done'); + await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining'); await expect(browser).toHaveURL('/todos'); }); @@ -37,17 +45,64 @@ describe('todos', { tags: ['todos'] }, () => { test('ignores an empty submission', async ({ screen }) => { // An exact interaction: the empty submit is the point of the test. await screen.getByRole('button', 'Add').tap(); + await expect(screen.getByRole('listitem')).toHaveCount(0); }); }); ``` -The agent does the flow; `expect` pins what must be true after each goal, -and that check lets the replay cache rerun the step later. `screen` actions -are for exact interactions and values. Files match the config `tests` glob, -default `tests/**/*.e2e.ts`. In a browser every test starts from a fresh -context with no page open, so it calls `app.open()` first; on a device a test -that skips `app.open()` starts where the previous test left the app. +The agent does the flow; `expect` pins what must be true at the end, and +that check lets the replay cache rerun every goal before it later. `screen` +actions are for exact interactions and values. Files match the config +`tests` glob, default `tests/**/*.e2e.ts`. In a browser every test starts +from a fresh context with no page open, so it calls `app.open()` first; on a +device a test that skips `app.open()` starts where the previous test left +the app. + +## Structure: one describe, arrange-act-assert, one behavior + +Write every test in this shape. When a test fails, its title alone should +say what broke, with no debugging needed, unless the arrange or act steps +themselves failed or the app changed under them. + +- **One top-level `describe` per unit or feature.** Name it after the + feature (`describe('todos')`, `describe('checkout')`) and put every test + for it inside, with the hooks those tests share. A file holds one feature. + Nest a `describe` only to share a hook or an option (a `session`, + `agentContext`, `tags`) with some of its tests. Setup tests are the + exception: `test.setup` must be top-level, outside any `describe`. +- **Arrange, act, assert, in that order**, with a blank line between the + three blocks: + - Arrange reaches the state the behavior starts from: `session`, + `app.open` on the screen's path, data from a fixture, and a shared + `beforeEach`. When the app offers no shorter way, an `agent.act` or + `screen` lines can arrange too, as in `marks a todo as done` above. + End such an arrange block with a wait for the state it reached + (`locator.waitFor()`, `agent.waitFor`). A setup goal that passed + without doing its job then fails there, not at the outcome check, and + the wait confirms the setup goals for the replay cache. + - Act does the one thing under test: one `agent.act` goal, or the + `screen` lines of one interaction. A wait for the state the action + triggers (`agent.waitFor`, `locator.waitFor`) closes the act block. + - Assert holds every `expect`, `expect.poll`, and `agent.assert`, and + nothing after them. No check sits between arrange and act steps. A + setup test's `session.save` is bookkeeping and comes after its check. +- **One behavior per test.** A test verifies one outcome, and its title + names it: `'marks a todo as done'`, not `'todos work'` or `'adds and + completes a todo'`. An "and" in a title usually means two tests. Several + assertions are fine when they all check that one outcome (the count and + the remaining label after an add). A second behavior is a second test. + When it starts from the first one's result, its arrange block repeats the + first one's act (or seeds the result through the API) rather than + chaining both into one test. + +For the replay cache (topic `agent`), a passing locator or engine matcher, +`locator.waitFor`, `agent.assert`, or `agent.waitFor` confirms every +`agent.act` before it; `expect.poll` and a plain-value `expect` confirm +nothing, so end with a locator check too. When the test fails, every goal +after the last passing confirmation runs live again on the next run: the +arrange block's closing wait keeps the setup goals cached while you fix the +act or the assertions. ## Registration @@ -356,15 +411,18 @@ test.setup('authenticate as admin', { sessions: ['admin'] }, async ({ app, scree ```ts // tests/dashboard.e2e.ts -import { test, expect } from 'e2e'; +import { describe, test, expect } from 'e2e'; -test('the dashboard opens directly', { session: 'admin' }, async ({ app, screen }) => { - await app.open('/dashboard'); - await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); +describe('dashboard', { session: 'admin' }, () => { + test('opens directly when signed in', async ({ app, screen }) => { + await app.open('/dashboard'); + + await expect(screen.getByRole('heading', 'Dashboard')).toBeVisible(); + }); }); ``` -- Setup tests are top-level; exactly one setup saves a given name and the +- Setup tests are top-level, outside any `describe`; exactly one setup saves a given name and the body saves every declared name once (`SESSION_CONTRACT` otherwise). Names match `[A-Za-z0-9_.-]{1,128}`. - A session holds cookies, local storage, and IndexedDB for one run, @@ -430,7 +488,7 @@ test as close to its screen as the app allows: `call`. ```ts -import { test } from '@e2e-dev/web'; +import { describe, test } from '@e2e-dev/web'; import { expect } from 'e2e'; import { z } from 'zod'; @@ -446,11 +504,15 @@ const projectTest = test.extend<{ project: { id: string } }>({ }, }); -projectTest('renames a project', { session: 'admin' }, async ({ app, screen, project }) => { - await app.open(`/projects/${project.id}/settings`); // straight to the screen under test - await screen.getByLabel('Name').fill('Atlas'); - await screen.getByRole('button', 'Save').tap(); - await expect(screen.getByRole('status')).toHaveText('Saved'); +describe('project settings', { session: 'admin' }, () => { + projectTest('renames a project', async ({ app, screen, project }) => { + await app.open(`/projects/${project.id}/settings`); // straight to the screen under test + + await screen.getByLabel('Name').fill('Atlas'); + await screen.getByRole('button', 'Save').tap(); + + await expect(screen.getByRole('status')).toHaveText('Saved'); + }); }); ``` @@ -460,7 +522,9 @@ in the app. Grow a test that spans several screens one screen at a time when you have not checked its steps live: write up to the next screen you have not seen -pass, run it, then add the next. A passing locator step takes milliseconds, +pass, end it with a temporary check of that screen, run it, then replace +the check with the next part. The finished test keeps only its outcome +checks, in the assert block. A passing locator step takes milliseconds, and an `agent.act` that a later check confirmed replays from the cache with no model call. The cache keeps those entries when a later line fails, so each run pays model calls only for its new part. That holds for a first diff --git a/skills/migrate-to-e2e/SKILL.md b/skills/migrate-to-e2e/SKILL.md index 5e35781ece..8e8938ef20 100644 --- a/skills/migrate-to-e2e/SKILL.md +++ b/skills/migrate-to-e2e/SKILL.md @@ -194,13 +194,33 @@ fixture (topic `writing-tests`). Record each in the progress file. Then each test, one at a time: -- Keep its title, and open the file with a comment naming the source +- Open the file with a comment naming the source (`// Ported from __e2e__/flows/login.yml`), so reports and reviewers map one to the other. - Translate literally first with the guide's tables, one awaited call per source step. Each source assertion maps to its own e2e assertion. A wait that fails on timeout (`waitForFunction`) becomes one more, so the e2e count is at least the inventory's. +- Then, before the first run, arrange the translated lines in the shape the + e2e skill's `writing-tests` topic gives ("Structure"). Lines move, split + tests copy the steps they need, and closing waits are added; no source + step or assertion is dropped: + - One top-level `describe` per source file or feature, named after it. + Setup tests stay top-level. + - A source test that checks one behavior keeps its title. A source test + that checks several (a check, more steps, another check) splits into + one test per behavior, each titled with its outcome; the file comment + names the source test each came from. Each split test repeats the + steps before its behavior in its own arrange block, or reaches that + state through a `beforeEach`, a fixture, or the app's API. + - Each test runs arrange, act, assert: the steps that reach the starting + state, the steps of the one behavior, then its assertions, last. + - A source assertion that only guarded a precondition (the dashboard + showed before the real steps began) becomes the wait that closes the + arrange block (`locator.waitFor()`, `agent.waitFor`). It still counts + toward the assertion count, since it fails on timeout. + - Splitting adds tests, never removes a check: the e2e count across the + split tests is still at least the source's. - A test the source skips is ported in full and stays skipped, with the source's reason, so it can be turned back on in either suite. Its row says so; when the reason allows, run it unskipped once and note the @@ -238,7 +258,9 @@ Then each test, one at a time: `agent.act` comes after the literal port passes, never instead of it, and only with a model configured. Propose it for a stretch the source kept patching: coordinate taps, picker spins, long tap sequences through a -wizard. Pin the outcome right after with a locator assertion (topic `agent`). +wizard. Keep the test's shape: an `agent.act` in the arrange block ends it +with a wait for the state it reached, and one under test is checked by the +locator assertions at the end (topic `agent`). ## 6. Prove each port