Repository navigation
feat(cli): e2e run --step holds each step until a key - #1540
Conversation
🟢 e2e web: 91 passedAll 91 tests in 41 files
Tested by e2e 0.22.0 · 1m 1s · web · run artifacts |
🟢 e2e web agent: 33 passed34 agent steps · 26 replayed from cache · 20 model calls · 87.7k tokens (85% cached) · $0.0053 All 33 tests in 5 files
Tested by e2e 0.22.0 · 1m 58s · web · run artifacts |
There was a problem hiding this comment.
All reported issues were addressed across 35 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
a93e515 to
d904b4b
Compare
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 12 replayed from cache · 66 model calls · 438.9k tokens (81% cached) · $0.0307 All 22 tests in 1 file
Tested by e2e 0.22.0 · 9m 51s · ios-simulator · run artifacts |
🟢 e2e android agent: 1 flaky, 19 passed24 agent steps · 15 replayed from cache · 50 model calls
|
| Test | Agent | Time | |
|---|---|---|---|
| tests-agent/scenarios.e2e.ts · 1 flaky, 19 passed | 24 steps · 50 calls | 5m 38s | |
| 🟢 | act completes Login Form | 1 step | 22.8s |
| 🟢 | act completes Infinite Scroll List | 1 step | 22.6s |
| act completes Modal Flow (1 failed attempt first) | 1 step | 12.2s | |
| 🟢 | act completes Bottom Tabs | 1 step | 7.0s |
| 🟢 | act completes Text Input Variations | 1 step | 6.5s |
| 🟢 | act completes Broken Accessibility | 2 steps · 5 calls | 15.4s |
| 🟢 | act completes Vision Only | 2 steps · 1 call | 6.7s |
| 🟢 | act completes Gestures | 1 step · 10 calls | 27.8s |
| 🟢 | act completes Async States | 1 step · 22 calls | 50.6s |
| 🟢 | act completes Debounced Search | 1 step | 6.9s |
| 🟢 | act completes Huge Virtualized List | 1 step | 1m 3s |
| 🟢 | act completes Flattened Registration Form | 2 steps · 1 call | 9.1s |
| 🟢 | act completes Flattened Login | 2 steps · 6 calls | 21.9s |
| 🟢 | act completes Sticky Chrome Target | 1 step | 7.4s |
| 🟢 | act completes WebView Accessibility | 1 step · 5 calls | 15.9s |
| 🟢 | act completes Permission Prompt | 1 step | 6.4s |
| 🟢 | act completes Product Catalog | 1 step | 8.3s |
| 🟢 | act completes Error Recovery | 1 step | 8.8s |
| 🟢 | act completes Choice Controls | 1 step | 8.5s |
| 🟢 | act completes Sequential Onboarding | 1 step | 9.6s |
Tested by e2e 0.22.0 · 5m 45s · android-emulator · run artifacts
🟢 e2e ios: 55 passed, 1 skippedAll 56 tests in 7 files
Tested by e2e 0.22.0 · 5m 46s · ios-simulator · run artifacts |
🟢 e2e android: 57 passed, 1 skippedAll 58 tests in 7 files
Tested by e2e 0.22.0 · 7m 52s · android-emulator · run artifacts |
A person can now step through a test: --step runs the first step, then holds each later top-level step until Enter, with the --watch keys (n to the next test, a for the rest, h for help, q to quit). The test clock stops while a step is held, so held time counts toward no timeout.
d904b4b to
c5cc552
Compare
|
lgtm! |
e2e runhad no way to stop between steps.--headedshows the app, but the test runs straight through. Playwright users reach forpage.pause()or--debugand find nothing.e2e run --stepruns the first step, then holds each later top-level step until a key is pressed. The keys follow--watch.nahq, EscHow it works
sequenceDiagram participant W as worker (StepRecorder) participant S as scheduler participant G as StepGate (CLI) W->>W: AttemptHolds stops the test clock W->>S: step-hold S->>G: beforeStep(step, { signal, show }) G-->>S: resolves on Enter / n / a S->>W: step-release W->>W: clock runs again with the time it had leftRunOptions.beforeStep(packages/e2e/src/run/step-hold.ts) is an internal seam. It is not public API, so there are no.d.tsor wire schema changes. A worker holds each top-level step over IPC (step-hold,step-release,step-cancel).Held time counts toward no timeout.
AttemptHoldsstops the test clock and rearms the deadline with what was left. The scheduler pushesmaxDurationback by the held time.Setup tests, retries, teardown steps, and steps inside an
expect.pollread run without stopping.Steps held together in one
Promise.allshare one prompt and run on one Enter. AwaitForResponseplaced beside the click that triggers it (the pattern indocs/browser.mdx) would otherwise run alone and time out while the click waits for its key.The prompt is drawn in the list reporter's live window under its test. Without a live window, it is one plain line.
--stepis refused, with exit 2:--watch;--workersabove 1;It forces one worker.
Docs:
docs/debugging.mdxgets a "Step through a test" section.page.pause(),--debug), and the skill references are updated.Issue
Closes #1251
Verified
Ran it locally: yes
e2ebuild in a scratch project. It has a one-page app and two test files. Each run was driven in tmux on LBox withe2e run --step.hshows the full key list andhagain hides it.nfinishes the test and holds at step 1 of the next test.aruns to the end. Result: 2 passed, exit 0.Promise.alltest, 7 s pause: both steps show in one prompt. I waited 7 s before pressing Enter, longer than the 5 swaitForURLgets. The test passed with exit 0.tests/integration/step-hold.test.tschecks against real workers that:interrupted;tests/unit/cli-step.test.tschecks the keys, the 150 ms guard, and the sharedPromise.allprompt.--step --headedon a Mac. That was before thePromise.allgrouping change.Reviewed with sol-loop: 1 round with gpt-6.1-sol. Sol verdict: approved. That approval was before the
Promise.allgrouping, which a later fresh-context review asked for.Written by Claude Opus 5.5 in Claude Code (T3 Code).