Skip to content

docs(skills): write tests as one describe per feature, arrange-act-assert, one behavior each - #1694

Merged
okwasniewski merged 5 commits into
mainfrom
kacper/eng-1267-skill-test-structure
Nov 11, 2026
Merged

okwasniewski merged 5 commits into
mainfrom
kacper/eng-1267-skill-test-structure

Conversation

@kacperkapusciak

@kacperkapusciak kacperkapusciak commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

The e2e skill told agents to check each agent.act outcome right after it. That produced long tests that chain several behaviors with checks in between, so a failing title said little about what broke. The skill now writes tests in one shape:

  • One top-level describe per unit or feature (test.setup stays top-level, since the collector rejects it inside a describe).
  • Arrange, act, assert, separated by blank lines, every expect / agent.assert at the end.
  • One behavior per test, named by its title. Several assertions are fine when they check that one outcome; a second behavior is a second test that repeats the setup in its arrange block.
  • An arrange block built from agent.act goals ends with a wait for its state (locator.waitFor(), agent.waitFor). It is not an assertion, but it fails a broken precondition where it broke, and it confirms the setup goals for the replay cache, so a failing assertion no longer evicts them (settleStagedTraces in packages/e2e/src/cache/context.ts keeps only entries before the last confirming step).
describe('todos', () => {
  beforeEach(async ({ app }) => {
    await app.open('/todos');
  });

  test('marks a todo as done', async ({ agent, screen }) => {
    await agent.act('add a todo named {title}', { params: { title: 'Write the release notes' } });
    await screen.getByRole('checkbox', 'Write the release notes').waitFor();

    await agent.act('mark the todo as done');

    await expect(screen.getByRole('status', 'Remaining')).toHaveText('0 remaining');
  });
});

Where it lands:

  • skills/e2e/references/writing-tests.md: new "Structure" section; every example reshaped.
  • skills/e2e/SKILL.md: quick start, the writing workflow, the rules (naming every check that confirms a recording: locator and engine matchers, agent.assert, and the waits), and the rerun advice.
  • agent.md, mcp.md (the live walk splits a second behavior into a second test; setup tests stay top-level), bug-bash.md (the repro closes its setup with an agent.waitFor, so a broken sign-in is not confirmed as the defect), debugging.md.
  • skills/migrate-to-e2e: a port still translates each source step literally, then arranges the lines in this shape before the first run. A source test that checks several behaviors splits into one test per behavior; a source assertion that only guarded a precondition becomes the wait closing the arrange block. Every source assertion still lands in some test, so the strictness count holds.
  • docs/core-concepts.mdx (opening example and the goals section), docs/cache.mdx, docs/self-healing.mdx: the "check after each act" advice, so the docs agree with the skill they ship with.
  • plugins/e2e/ regenerated; patch changeset for e2e (migrate-to-e2e is not shipped in the package).

Verified

Ran it locally: yes

  • A fresh agent with no context read only SKILL.md and writing-tests.md and wrote a deterministic todos suite for the testbed app. It produced one describe('todos'), four single-behavior tests, and arrange/act/assert blocks with every expect last. Run with the built CLI against the testbed app (a throwaway config on port 4299, since another testbed held 4271):
 ✓ target "web" command at http://127.0.0.1:4299/ ready 89ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > adds a todo to the list 1.75s
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > marks a todo as done 170ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > deletes a todo from the list 231ms
   ✓ |web| tests/tmp-verify/todos.e2e.ts > todos > shows only completed todos under the Done filter 264ms
 Test Files  1 passed (1)
      Tests  4 passed (4)
  • A fresh agent with only the migrate skill's step 5, the Structure section, and the Playwright guide ported a one-test Playwright checkout flow (5 assertions, checks between steps). It produced one describe('checkout') with three single-behavior tests (adds a book to the cart, updates the total when quantity changes, confirms the order at checkout), turned the Shop heading check into a beforeEach wait, and mapped every source assertion to a final assertion. Not run against an app.
  • The cache claims were checked against packages/e2e/src/cache/context.ts and run/steps.ts in a fresh-context review; session, tags, and agentContext on a describe against collect/registry.ts.
  • Not run: the agentic examples (no model call was made for this change).

@github-actions github-actions Bot added the size:XL Up to 999 changed lines, not counting tests, fixtures, translated docs, or the lockfile label Nov 11, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 16 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/core-concepts.mdx Resolved
Comment thread skills/e2e/references/mcp.md Resolved
Comment thread skills/e2e/SKILL.md Resolved
Comment thread plugins/e2e/skills/e2e/SKILL.md Resolved
Comment thread docs/core-concepts.mdx Resolved
Comment thread docs/cache.mdx Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 10 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread skills/migrate-to-e2e/SKILL.md Resolved

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0 issues found across 2 files (changes from recent commits).

View guided diff | Turn on auto-fix | Re-trigger cubic

…-test-structure

# Conflicts:
#	plugins/e2e/skills/e2e/references/mcp.md
#	skills/e2e/references/mcp.md
@okwasniewski
okwasniewski merged commit dc01aaa into main Nov 11, 2026
30 checks passed
@okwasniewski
okwasniewski deleted the kacper/eng-1267-skill-test-structure branch November 11, 2026 18:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:XL Up to 999 changed lines, not counting tests, fixtures, translated docs, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants