Repository navigation
Conversation
There was a problem hiding this comment.
All reported issues were addressed across 11 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
🟢 e2e android agent: 20 passed24 agent steps · 15 replayed from cache · 45 model calls · 293.0k tokens (86% cached) · $0.0177 All 20 tests in 1 file
Tested by e2e 0.24.0 · 5m 36s · android-emulator · run artifacts |
🟢 e2e ios: 59 passed, 1 skippedAll 60 tests in 7 files
Tested by e2e 0.24.0 · 6m 45s · ios-simulator · run artifacts |
🟢 e2e android: 60 passed, 1 skippedAll 61 tests in 7 files
Tested by e2e 0.24.0 · 8m 33s · android-emulator · run artifacts |
🟢 e2e ios agent: 21 passed, 1 skipped25 agent steps · 12 replayed from cache · 63 model calls · 453.5k tokens (84% cached) · $0.0285 All 22 tests in 1 file
Tested by e2e 0.24.0 · 8m 42s · ios-simulator · run artifacts |
|
[claude-sonnet-5-5] responding on behalf of kdy1 Left for the human:
|
1240b61 to
3789edd
Compare
3789edd to
c255a6c
Compare
c255a6c to
de86937
Compare
e55294b to
98e2437
Compare
d1c8a0c to
234bd02
Compare
okwasniewski
left a comment
There was a problem hiding this comment.
I re-ran the comparison on current main, since the numbers in the description are from agent-device 0.21.36 and a main that predates #1668.
Setup
- main
4a48dc253against this branch merged with main, both on agent-device 0.21.41. - Two iOS 26.5 simulators, one build on each, run at the same time, then swapped.
- The scratch test from the description (21 x
scrollUntilVisiblethen tap on the huge virtualized list). - A scratch copy of the benchmark app shows the last tapped row on screen, so each tap has an outcome: it either reached the row or it did not. I counted that instead of "the last two reads disagree", which is the wait's own exit condition.
Result
| Scroll-then-tap pairs | main | this branch |
|---|---|---|
| Idle, taps lost | 0 / 420 | 0 / 420 |
| 15 CPU hogs, taps lost | 0 / 252 | 0 / 252 |
transition: 0, taps lost |
0 / 63 | 8 / 63 |
| Target scrolled off screen, tap fails | 1 | 1 |
| Scroll return to tap, median (idle) | ~1,755 ms | ~1,650 ms |
Main already loses no taps here. The re-read from #1668 covers the moving list, so this PR is a refactor of that wait plus a new option, not a fix.
Before this can merge
transition: 0regresses. On this branch 0 means no rest check at all, so a tap right after a scroll lands on a list that still moves: 8 of 63 taps were swallowed, all on slow flings, about 230 ms afterscrollreturned. Main keeps its re-read at 0 and lost none. Please keep the rest check when the budget is 0.- A look-alike without bounds never counts as at rest.
sighting()returnsundefinedwhen any look-alike has norect, andsameSightingis then always false, so the loop runs to the cap. A scratch unit test (still screen,gesture: 500, one extra same-named button with no rect) took 4000 ms and 72 reads, against 500 ms and 2 reads when that node has a rect. Main treats missing bounds as unmoved. Please leave such look-alikes out of the sighting and add a test. - The title and description should say what the PR is now: the same protection as main, one read and about 100 ms cheaper per scroll-then-tap, and
transitionby kind of action.
Worth a decision
- At the cap, or when the action's timeout cannot fit another read, the action now goes ahead on the latest read. Main failed with
OPERATION_TIMEOUTand sent nothing. That is a silent tap on a control last seen moving. - The
TransitionBudgetsoption is new public API. I would split it into its own PR so it can be judged on its own. docs/reference/mobile.mdxgrows by 81 lines for one option and explains look-alikes and read pairs. Users need the object shape, thatgestureis a minimum, and the 4 s cap.
The failure that is left
One run on each build failed the same way: scrollUntilVisible returned on a row it saw in the middle of a fling, the fling carried the row off screen, and the tap failed after 30 s with locator matched no nodes. The tap wait cannot fix that. scrollUntilVisible should wait for the list to stop before it reports the row visible. That looks like the fix worth making for sticky chrome: accept below the fold.
Limits of my run: iOS simulator only, one scenario, no Android, no CI. If you have a case where current main still loses a tap, please share it, that would change this.
|
[claude-sonnet-5-5] responding on behalf of kdy1 Pushed 0f560a7 for items 1 to 3 of the changes-requested review:
Checks: the 632 mobile unit tests, Left for the owner, not changed here: whether an action that cannot fit another read should go ahead on the latest read or fail with |
`mobile({ transition })` takes a number, the same wait after every action, or
`{ default, gesture, launch }` (new exported type `TransitionBudgets`). The
engine records the kind of the last action: `gesture` for a root scroll, node
swipe, or coordinate swipeTo, `launch` after a launch, relaunch, close, or
openLink. A scroll can return while its content moves and needs a longer wait.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…moving After a scroll or swipe the `gesture` budget is only the least wait. The engine then reads until two sightings of the control and its look-alikes agree, at most 4 s after the gesture, and that read is also the relocation. `scroll` can return while a fling still moves the list, and a tap aimed at a moving row lands on the row passing under it. A budget of 0 is still no wait, and an action whose timeout cannot fit another read goes ahead on the latest one. The default and launch paths and the agent are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A fill or clear after a scroll or swipe now takes the read that found the field at rest as its re-read, so it costs no read of its own and a short timeout does not fail it where a tap goes ahead on the latest read. The checks `relocated` makes for an input move into one method both reads use. The changeset and docs name the cap (4 s, or the `gesture` budget when that is larger) and the timeout exit, and the skill reference lists the link case of `launch`. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The rest wait stopped reading once the action's timeout could not fit another read and its pause, but a long press also needs its hold left, or `InputBudget.send` refuses it unsent. The hold is now part of that exit condition, so a long press after a scroll goes ahead on the latest read. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ook-alikes with no bounds A gesture budget of 0 skipped the rest check as well as the sleep, so a tap right after a scroll could land on a list that still moves; main kept its re-read at 0 and lost no tap. A 0 budget now sleeps for nothing and the control is still read until it stops moving. A look-alike with no bounds made the sighting undefined, so two reads never agreed and the wait ran to its 4 s cap. Such a node is now left out of the sighting, as main treats missing bounds as unmoved. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
0f560a7 to
4b0d7b9
Compare
okwasniewski
left a comment
There was a problem hiding this comment.
Re-ran on 4b0d7b9e17. The three items are fixed. With transition: 0 the branch swallowed 0 of 126 scroll-then-tap pairs on the iOS simulator (8 of 63 before), and the 16 slow flings waited about 2 s for rest. The three new unit tests fail on the previous head.
One more thing before merge. After a gesture, waitForTransition returns at the samePlace shortcut before it reaches atRest. On main steadied ran after that shortcut, so main still re-read. With repeated rows the shortcut fires whenever the tapped row sits where a look-alike sat before the scroll. Scratch test: rows named Add at y 200, 300, 400, 500 before the scroll, 300, 400, 500, 600 in the read that resolves the tap, still moving after. Main takes 4 reads and taps at rest. This branch takes 1 read and taps at once, without the gesture floor. Please skip the shortcut after a gesture when the control has look-alikes, and add that test. The earlier thread on this said the shortcut predates the PR. It does, but the re-read behind it does not survive here.
Decisions:
- Keep
OPERATION_TIMEOUTwhen the action's timeout cannot fit another read, as main does. Going ahead at the cap is fine. - Take
TransitionBudgetsout of this PR. At a floor of 0 the rest check alone lost no tap, so I see no case for a separategesturebudget yet, and none forlaunch. - Cut the docs to the shape users need:
gestureis a minimum, the wait ends when the control stops or after 4 s, and 0 still checks for rest.
The description still says the device suites were not rerun on this head. CI ran them green.
… look-alikes The "same place" shortcut in waitForTransition returned before the control was read at rest. With repeated rows, a row can sit where a look-alike sat before the scroll, so the shortcut fired while the list was still moving and the tap went out without the gesture floor. Main still re-read there. After a gesture the shortcut now holds only for a control with no look-alikes, in the read before the action or the read it resolved on. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
[claude-sonnet-5-5] responding on behalf of kdy1 Pushed 686267d for the Not changed here, because each one changes the PR's public shape and is the owner's call: removing |
gesturebudget is the least wait, then the engine reads until two sightings of the control and its look-alikes agree, at most 4 s after the gesture. That read is also the relocation read, so a list that already sits still costs no extra read. A look-alike with no bounds is left out of the sighting.steadied), which compared the control's own bounds, 100 ms apart, for up to 2 s after the gesture and failed the action withOPERATION_TIMEOUTwhen its budget ran out first. This one compares the control with its look-alikes, 50 ms apart, runs up to 4 s, and goes ahead on the latest read when the budget runs out. perf(mobile): skip the post-gesture stabilization wait for a test's swipe and scroll #1668's opt-out of agent-device's own post-gesture settle (postGestureStabilization: false) stays as it is.mobile({ transition })also takes{ default, gesture, launch }(newTransitionBudgetstype), so the least wait after a gesture can differ from the wait after a tap. The wait fix(mobile): wait for transitions before screen swipes #1564 puts before a screen swipe follows the same kinds. A number still sets one wait for every action. A budget of 0 is no wait at all after any other action; after a scroll or swipe it sleeps for nothing and the rest check still runs.OPERATION_TIMEOUT. Agent actions and the default and launch paths are unchanged. This is a behavior change from main and is open for the owner to decide.flowchart LR A[scroll or swipe returns] --> B[sleep the gesture budget] B --> C[read the screen] C --> D{two sightings agree?} D -- no --> E{cap or timeout reached?} E -- no --> C D -- yes --> F[act on the control as read] E -- yes --> FVerified
Latest head (rest check kept at a budget of 0, look-alikes with no bounds left out): the 632 mobile unit tests pass and the three new or changed tests fail on the previous head;
tsc,oxlint,plugins:check, anddocs:checkpass. The device suites below were not rerun on this head.Ran it locally: yes
apps/mobile-benchmarkthrough the built CLI on an iOS simulator and an Android emulator,@e2e-dev/mobilebuilt from main and from this branch, both on the pinned agent-device 0.21.36 (--workers 1 --retries 0, one run at a time). Both builds carried a wrapper around the agent-device client that records every call (not in this PR).scrollUntilVisiblethen tap on the huge virtualized list,--repeat-each 10, two runs per build in the order main, branch, branch, main (main was f6aba97, before fix(mobile): wait for transitions before screen swipes #1564 and perf(mobile): skip the post-gesture stabilization wait for a test's swipe and scroll #1668 landed). That is 840 scroll-then-tap pairs, and all 40 repeats pass. A tap counts as sent on a moving list when the last two snapshots before it disagree about where the rows sit (more than 1 px).Fisher exact, one-sided: p = 0.015. Five of main's six were in one run (the other main run had one), so without that run the contrast alone is 1/210 against 0/420. Slow flings are 16 of 840 scrolls (1.9%). On the branch each waited 2.0 to 2.5 s for two reads to agree, with 3 to 5 reads, and none reached the 4 s cap. The baseline is main before #1668, with agent-device's own settle in place. The table was not repeated against main with #1668's re-read, so it shows what the rest wait adds over that settle, not over
steadied.pnpm checkpass. The previous head (33388e3, the PR's four commits on f64e7db) passed CI with all four device jobs (iOS and Android, deterministic and agentic) green. Locally, the iOS deterministic suite on a simulator with 0.21.37 gave 57 passed, 1 skipped, 1 failed (672 s, 1-minute load 3.5 at start after the 4-minute wait cap). The failure isopenLink opens the scenario a deep link names, new from test(mobile-benchmark): cover device.openLink with a real deep link #1805: the app installed on that simulator is the 11-03 build, whose bundle has noe2e-benchmark://routing, so the link opens and the app ignores it. CI builds or repacks the app from each head's own source. Android was not run locally.Not covered: this branch against main with #1668's re-read on 0.21.37, a swipe that starts a UIKit transition (the tree reports that screen's final frame at once, so only the
gesturebudget protects the tap, as before), and real devices.The scratch test
🤖 Generated with Claude Code