Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/skill-test-structure.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"e2e": patch
---

The bundled e2e skill now writes tests with one top-level `describe` per feature, one behavior per test, and arrange, act, assert order with every assertion at the end, instead of a check after each `agent.act`.
15 changes: 10 additions & 5 deletions docs/cache.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,19 @@ run summary counts the outcomes:

## When a recording is written

Follow each `act` with a check of its outcome. In `read-write` mode, the
runner saves the recording at the end of the attempt when a later
verification step passed.
End the test with a check of its outcome. In `read-write` mode, the runner
saves the recording of each `act` at the end of the attempt when a later
verification step passed. One check at the end confirms every `act` before
it.

```ts title="tests/cart.e2e.ts"
import { expect, test } from 'e2e';

test('adds an item', async ({ app, agent, screen }) => {
await app.open('/cart');

await agent.act('add one item to the cart');

await expect(screen.getByRole('status')).toHaveText('1 item');
});
```
Expand All @@ -53,15 +56,17 @@ when the attempt ended:
| --- | --- |
| `saved` | Recorded for the next run. |
| `kept` | The entry stays as it is. |
| `unconfirmed` | Not saved: no check passed after the step. Follow it with a locator assertion or `agent.assert`. |
| `unconfirmed` | Not saved: no check passed after the step. End the test with a locator or engine assertion or `agent.assert`, or wait for the state with `locator.waitFor()` or `agent.waitFor`. |
| `evicted` | The entry was deleted: its replay ended in a failure or a repair, or no check confirmed it before the attempt failed. |
| `no-change` | Nothing recorded: no control appeared, went away, or changed state, and the route stayed the same. That includes a step whose only change is inside a [frame from another site](/browser#reach-into-an-iframe). Such a step runs live on every run. |
| `not-written` | Not saved: the store rejected the write, or refused the recording, for example one over the file store's size limit. |

A step that took no action records nothing and reports no `write`.

A failing attempt still saves the steps verified before the failure. It
deletes unverified entries and the replays involved in the failure. A custom store needs a `delete` method for this. Cancelling
deletes unverified entries and the replays involved in the failure. To keep
setup goals recorded while you fix a failing assertion, end the setup with
a wait for its state, such as `locator.waitFor()`. A custom store needs a `delete` method for this. Cancelling
a run leaves the cache unchanged.

<Accordion title="Failures where no model answered">
Expand Down
48 changes: 28 additions & 20 deletions docs/core-concepts.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,22 +16,26 @@ A test mixes three types of step:
With a [custom executor](/executors), `act` and `assert` need no model.

```ts title="tests/todos.e2e.ts"
import { test, expect } from 'e2e';
import { describe, test, expect } from 'e2e';

test('todos survive a filter round-trip', async ({ app, agent, screen }) => {
await app.open('/todos');
describe('todos', () => {
test('the Done filter shows only completed todos', async ({ app, agent, screen }) => {
await app.open('/todos');
await agent.act('add todos "Buy milk" and "Walk the dog"');
await agent.act('complete "Buy milk"');
await agent.waitFor('"Buy milk" is completed and "Walk the dog" is open');

await agent.act('add todos "Buy milk" and "Walk the dog"');
await expect(screen.getByTestId('todo')).toHaveCount(2);
await screen.getByRole('tab', 'Done').tap();

await agent.act('complete "Buy milk"');
await agent.assert('only one todo remains open');

await screen.getByRole('tab', 'Done').tap();
await expect(screen.getByTestId('todo')).toHaveCount(1);
await expect(screen.getByTestId('todo')).toHaveCount(1);
await agent.assert('the only todo listed is "Buy milk"');
});
});
```

Each test checks one behavior in three blocks: arrange the starting state,
act on the behavior, then assert the result.

Await each agent call, screen call, and `expect.poll`. If a test returns
while a step still runs, the test fails with `STEP_NOT_AWAITED`.

Expand All @@ -55,26 +59,30 @@ await agent.act('open the Pro plan, set the seats to {seats}, and check out', {
});
```

```ts title="Good: one atomic goal per call, each checked before the next"
await agent.act('open the Pro plan');
await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible();
```ts title="Good: one atomic goal per call, arrange, act, assert"
test('checks out the Pro plan', async ({ app, agent, screen }) => {
await app.open('/plans');
await agent.act('open the Pro plan');
await agent.act('set the seats to {seats}', { params: { seats: 3 } });
await agent.waitFor('the cart shows the Pro plan with 3 seats');

await agent.act('set the seats to {seats}', { params: { seats: 3 } });
await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats');
await agent.act('check out');

await agent.act('check out');
await expect(screen.getByText('Order confirmed')).toBeVisible();
await expect(screen.getByText('Order confirmed')).toBeVisible();
});
```

A failure then names the goal that broke. Each goal replays from its own
The setup goals come first and end with a wait for the state they reach.
The goal under test comes next. The assertions on its result come last. A
failure then names the goal that broke. Each goal replays from its own
recording and gets its own model-call and action
[budgets](/reference/agent#budgets). The actions toward one outcome are not
goals: opening the screen or menu that holds a control, scrolling to it,
closing a banner, or filling the fields of one form.

- You control the order of the goals. The agent picks the actions.
- Check each goal's result with an `expect`. The [replay cache](/cache)
records a goal only after a later check passes.
- End the test with an `expect` on the result. The [replay cache](/cache)
records a goal only after a later check or wait passes.
- Use the words that the screen shows.
- Give test data in `params`. For a password, give a `Secret`. The model
never gets its value as text. See [Authentication](/authentication).
Expand Down
2 changes: 1 addition & 1 deletion docs/self-healing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ clicks. If the discount button moved, the agent finds it. If discounts stop
working, the step or the check after it fails.

A goal can still pass in a way you didn't intend, for example with a
different discount. Follow each `act` with a deterministic check of the
different discount. End the test with a deterministic check of the
result, such as the `toHaveText('10% off')` above. That check runs on every
run, replayed or healed.

Expand Down
56 changes: 38 additions & 18 deletions plugins/e2e/skills/e2e/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,14 +81,18 @@ export default {

```ts
// tests/billing.e2e.ts
import { test } from '@e2e-dev/web';
import { describe, test } from '@e2e-dev/web';
import { expect } from 'e2e';

test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => {
await app.open('/settings/billing');
await agent.act('upgrade the workspace to the Pro plan');
await expect(screen.getByRole('status')).toContainText('Pro');
await expect(browser).toHaveURL('/settings/billing');
describe('billing', () => {
test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => {
await app.open('/settings/billing');

await agent.act('upgrade the workspace to the Pro plan');

await expect(screen.getByRole('status')).toContainText('Pro');
await expect(browser).toHaveURL('/settings/billing');
});
});
```

Expand All @@ -106,13 +110,19 @@ test('a member upgrades to Pro', async ({ app, agent, screen, browser }) => {
path's actions stay out of the test. A `screen` line is the `Test line:`
its action printed, copied as is. When the outcome shows, run its
`expect` with `run_code` and add it once it passes; a passing statement
also joins `test_lines`. The walk ends with the test written. Without a
also joins `test_lines`; checks go in the test's assert block at the end.
The walk ends with the test written. Without a
server, learn the accessible names the app renders from the components or
a `--headed` run.
3. A step is an `agent.act` goal or `screen` lines. Drive the flow with
`agent.act`, one atomic goal per call, and pin each outcome right after
with `expect` or `agent.assert`. A test with three goals makes three
calls. Start the test on the screen it covers, since every rerun pays
3. Shape the file before the steps (topic `writing-tests`, "Structure"):
one top-level `describe` per feature, and in it one test per behavior,
titled with the outcome it verifies. Each test runs arrange, act, assert,
in that order: reach the starting state, do the one thing under test,
then every `expect` or `agent.assert` at the end. A second outcome worth
checking is a second test, so a failing title says what broke.
A step is an `agent.act` goal or `screen` lines. Drive the flow with
`agent.act`, one atomic goal per call; three goals make three calls.
Start the test on the screen it covers, since every rerun pays
for the steps before: in a browser a saved session instead of the
sign-in form and `app.open` on that screen's path, on a device
`device.openLink`, and data from the app's API (topic `writing-tests`).
Expand Down Expand Up @@ -181,7 +191,8 @@ lines you changed:
call's own `{ timeout }` still wins. An `expect` already gives up after
5 s by default (`assertionTimeout`).
- Keep the replay cache on. A failed run keeps the recording of every goal
whose check passed, and the next run replays those with no model call.
a later check or wait confirmed (the wait closing the arrange block, topic
`writing-tests`), and the next run replays those with no model call.
New, changed, and failed goals run live, and so does a replay the trace
page shows as handed off. `agent.assert`, `agent.waitFor`, and
`agent.extract` always call the model. Leave `--no-cache` off: it sends every goal to the model
Expand Down Expand Up @@ -237,14 +248,23 @@ lines you changed:
`credentials.user(name).password` or `secrets.get(name)` (separate
namespaces: `secrets.get` never returns a password), and hand the opaque
`Secret` only to `fill()` or `agent.act` params.
- Tests: one top-level `describe` per feature, one behavior per test,
arrange, act, assert with every assertion last (topic `writing-tests`).
Bad: one test that adds a todo, checks it, marks it done, and checks
again. Good: `'adds a todo to the list'` and `'marks a todo as done'`,
the second adding its todo in its arrange block.
- Agent instructions: one goal per `act`, the wording on screen, real values
in params. A goal is atomic: one outcome that one check confirms. Two
outcomes worth checking are two goals, so split them into two `act` calls
(topic `agent`). Bad: `'add a todo and mark it done'`. Good: `'add a
todo'`, check, then `'mark the todo as done'`, check. Judge meaning, not
phrasing: `toContain('Pro')`, not an exact sentence a model produced.
- Check each agent goal's outcome. A passing `act` with a recorded check can
be cached and replayed without model calls (topic `agent`).
outcomes are two goals in two `act` calls (topic `agent`), never `'add a
todo and mark it done'`. Judge meaning, not phrasing: `toContain('Pro')`,
not an exact sentence a model produced.
- End every test that runs an agent goal with a verifying check: a
locator matcher (`expect(locator)`), an engine matcher
(`expect(browser).toHaveURL`), or `agent.assert`. A passing one confirms
every `act` before it, so they can be cached and replayed without model
calls, and so do `locator.waitFor`, `browser.waitForURL`, and
`agent.waitFor`. Reads, `expect.poll`, and plain-value `expect` confirm
nothing (topic `agent`, "The replay cache").
- Shape the agent for this app: `context` for vocabulary the screens use,
`system` for how it works, tools for a test API, named personas under
`agents`. When a step fails, tighten the goal first, then the context, then
Expand Down
38 changes: 24 additions & 14 deletions plugins/e2e/skills/e2e/references/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,9 @@ context, so each persona records and replays its own steps.
## act: one goal

```ts
import { credentials, expect } from 'e2e';
import { credentials } from 'e2e';

await agent.act('add a todo named "Buy milk"');
await expect(screen.getByRole('listitem', 'Buy milk')).toBeVisible();
await agent.act('mark the "Buy milk" todo as done');
await expect(screen.getByRole('checkbox', 'Buy milk')).toBeChecked();
await agent.act('invite {email} as an editor', { params: { email: 'ada@example.test' } });

const member = credentials.user('member');
Expand Down Expand Up @@ -295,33 +292,46 @@ elsewhere on the current screen.
- One goal per `act`; goal order is the test's, the path the model's.
- Keep each goal atomic: one outcome that one check confirms. When two
parts of an instruction each leave a state worth checking, they are two
goals: give each its own `act` and check. Steps toward a single outcome
goals, each its own `act`. Steps toward a single outcome
are the path, not more goals: scrolling to a control, closing a banner,
opening the menu that holds it, filling the fields of one form.
- A split flow fails at the goal that broke. Each goal replays from its own
recording and gets its own `maxSteps` and `maxModelCalls`; the test
timeout still caps them all.

```ts
// BAD: three goals in one instruction, one recording, no check between them.
// BAD: three goals in one instruction, one recording.
await agent.act('open the Pro plan, set the seats to {seats}, and check out', { params: { seats: 3 } });

// GOOD: three atomic goals, each pinned before the next.
await agent.act('open the Pro plan');
await expect(screen.getByRole('heading', 'Pro plan')).toBeVisible();
await agent.act('set the seats to {seats}', { params: { seats: 3 } });
await expect(screen.getByRole('status', 'Seats')).toHaveText('3 seats');
await agent.act('check out');
await expect(screen.getByText('Order confirmed')).toBeVisible();
// GOOD: arrange with two atomic goals, act with the one under test, assert last.
test('checks out the Pro plan', async ({ app, agent, screen }) => {
await app.open('/plans');
await agent.act('open the Pro plan');
await agent.act('set the seats to {seats}', { params: { seats: 3 } });
await agent.waitFor('the cart shows the Pro plan with 3 seats');

await agent.act('check out');

await expect(screen.getByText('Order confirmed')).toBeVisible();
await expect(screen.getByText('Pro plan, 3 seats')).toBeVisible();
});
```

Choosing the seats is its own behavior, so it gets its own test that ends
after `set the seats` and asserts the seat count. Tests follow arrange, act,
assert, one behavior each (topic `writing-tests`, "Structure").

- Use the words on screen: `'open the Billing tab'`, not `'upgrade'`.
- Values go in `params`, never expanded: `act('rename to {name}', { params })`.
- Do not describe mechanics the runner handles: waiting, scrolling, retries.
- Pin every `act` outcome right after it; that check lets the cache record:
- End the test with a check of the outcome. A passing locator or engine
matcher, `agent.assert`, or wait confirms every `act` before it, which
lets the cache record them; close an arrange block of `act` goals with a
wait for its state, so a failure later in the test keeps them recorded:

```ts
await agent.act('create a workspace named "Atlas" on the Pro plan');

await expect(screen.getByRole('status')).toHaveText('Created "Atlas" on the Pro plan');
```

Expand Down
31 changes: 20 additions & 11 deletions plugins/e2e/skills/e2e/references/bug-bash.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,21 +220,30 @@ verify one area after another.

```ts
// tests/bugbash/cart-line-quantity.e2e.ts
import { test } from '@e2e-dev/web';
import { describe, test } from '@e2e-dev/web';
import { credentials } from 'e2e';

test('a cart line counts its quantity', { tags: ['bugbash'] }, async ({ app, agent }) => {
const shopper = credentials.user('bb-cart');
await app.open('/');
await agent.act('sign in with the given credentials', {
params: { username: shopper.username, password: shopper.password },
describe('cart', { tags: ['bugbash'] }, () => {
test('a cart line totals its quantity times the unit price', async ({ app, agent }) => {
const shopper = credentials.user('bb-cart');
await app.open('/');
await agent.act('sign in with the given credentials', {
params: { username: shopper.username, password: shopper.password },
});
await agent.act('add "Dune" to the cart');
await agent.waitFor('you are signed in and the cart holds "Dune"');

await agent.act('set the quantity of "Dune" in the cart to 2');

await agent.assert('the cart line for "Dune" totals twice its unit price');
});
await agent.assert('the app shows you are signed in');
await agent.act('add "Dune" to the cart');
await agent.act('set the quantity of "Dune" in the cart to 2');
await agent.assert('the cart line for "Dune" totals twice its unit price');
});
```

Write it in the shape topic `writing-tests` describes: arrange, closed by
a wait for the starting state, then the one action that shows the defect,
then the assertions. The wait keeps a broken setup from passing as the
defect, and keeps the setup goals cached across the failing runs.
3. Run your repros with the bug-bash config and an output directory of your
own, `npx e2e run tests/bugbash/<slug>.e2e.ts --config
e2e.bugbash.config.ts --output .e2e/bugbash/verify-<area>` (a run clears
Expand All @@ -244,7 +253,7 @@ verify one area after another.
failed `act` whose message names the defect. A failed `act` carries the
code the model chose, so only its message and screenshot tell the defect
apart from a failure short of it. A blocked `act`, a timeout, a setup
error, or a failure that names something else means the repro is wrong:
error, a failed arrange wait, or a failure that names something else means the repro is wrong:
sharpen the step and rerun. A passing test means the bug did not
reproduce: reject the finding, say so, and move the test out of
`tests/bugbash/` (to `.e2e/bugbash/extra-tests/`, or offer it as a
Expand Down
Loading
Loading