Repository navigation
docs(skills): write tests as one describe per feature, arrange-act-assert, one behavior each - #1694
Merged
Merged
Conversation
…sert, one behavior each Refs ENG-1267
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 16 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
… reshape the core concepts example
…rrange-act-assert, one behavior each
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
…-test-structure # Conflicts: # plugins/e2e/skills/e2e/references/mcp.md # skills/e2e/references/mcp.md
2 of 5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The e2e skill told agents to check each
agent.actoutcome right after it. That produced long tests that chain several behaviors with checks in between, so a failing title said little about what broke. The skill now writes tests in one shape:describeper unit or feature (test.setupstays top-level, since the collector rejects it inside adescribe).expect/agent.assertat the end.agent.actgoals ends with a wait for its state (locator.waitFor(),agent.waitFor). It is not an assertion, but it fails a broken precondition where it broke, and it confirms the setup goals for the replay cache, so a failing assertion no longer evicts them (settleStagedTracesinpackages/e2e/src/cache/context.tskeeps only entries before the last confirming step).Where it lands:
skills/e2e/references/writing-tests.md: new "Structure" section; every example reshaped.skills/e2e/SKILL.md: quick start, the writing workflow, the rules (naming every check that confirms a recording: locator and engine matchers,agent.assert, and the waits), and the rerun advice.agent.md,mcp.md(the live walk splits a second behavior into a second test; setup tests stay top-level),bug-bash.md(the repro closes its setup with anagent.waitFor, so a broken sign-in is not confirmed as the defect),debugging.md.skills/migrate-to-e2e: a port still translates each source step literally, then arranges the lines in this shape before the first run. A source test that checks several behaviors splits into one test per behavior; a source assertion that only guarded a precondition becomes the wait closing the arrange block. Every source assertion still lands in some test, so the strictness count holds.docs/core-concepts.mdx(opening example and the goals section),docs/cache.mdx,docs/self-healing.mdx: the "check after each act" advice, so the docs agree with the skill they ship with.plugins/e2e/regenerated; patch changeset fore2e(migrate-to-e2eis not shipped in the package).Verified
Ran it locally: yes
SKILL.mdandwriting-tests.mdand wrote a deterministic todos suite for the testbed app. It produced onedescribe('todos'), four single-behavior tests, and arrange/act/assert blocks with everyexpectlast. Run with the built CLI against the testbed app (a throwaway config on port 4299, since another testbed held 4271):checkout flow(5 assertions, checks between steps). It produced onedescribe('checkout')with three single-behavior tests (adds a book to the cart,updates the total when quantity changes,confirms the order at checkout), turned theShopheading check into abeforeEachwait, and mapped every source assertion to a final assertion. Not run against an app.packages/e2e/src/cache/context.tsandrun/steps.tsin a fresh-context review;session,tags, andagentContexton adescribeagainstcollect/registry.ts.