Repository navigation
fix(agent): show the judge the current location - #1515
Conversation
The built-in judge never saw the current URL, so an assertion or wait about the address ended inconclusive or timed out whatever the address was. buildPrompt now puts the redacted location on <observation> as a `location` attribute: a path on the app's origin, the whole address off it, without userinfo, cut like a link target. waitFor also judges again when only the location changes, except under vision: 'only', which is not shown it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 18 files
Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.
View guided diff | Re-trigger cubic
…nly' skips A wait ignored a clock-shaped value only to keep a device's screen title from counting as a change; it also hid a real move such as /at/10:15 to /at/10:16. Only an http(s) address is now compared as it is, so a bare device title such as "Timer: 00:12" still ignores its clock. The docs and changeset say a location change alone does not wake a vision-only wait, the security page says an off-origin address reaches the judge whole, and the lines that list what a judgment sees name its params and the agent's context too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
okwasniewski
left a comment
There was a problem hiding this comment.
Automated review (agent-run thermo-nuclear review, a human maintainer reviews before merge).
Thanks, this is a real bug and a clean fix. Checked at 6f76a74: typecheck, lint, unit (3262) and integration agent/agent-vision/agent-policy pass. Redaction matches act (same observation.location, prompt redacted again by Invocation), no cache-key impact (judgments are never cached, act keys don't include prompts), mobile location (bundleId / title) behaves sanely. Your declines on the cubic threads hold up.
Two structural changes before this is ready:
- Compute the judge's location in
prepareObservation, notbuildPrompt.prepareObservationalready getsappOrigin(invocation.ts:244,observation-feed.ts:403). Store the judged location as its own field onAgentObservation(notpath, the replay cache'ssameRouteuses that). This deletesPromptInput.appOrigin(prompts.ts:63-64), theappOriginline ininvocation.ts:335and thejudgedLocationcall inbuildPrompt(prompts.ts:101), and letslocationShaperead the stored field instead of re-runningjudgedLocation(..., undefined)and re-parsing its own output withisHttpUrl(observation.ts:326-329). - One comparison in
waitForNextJudgment.index.ts:357,389-390adds a second shape plus atreeWithheldbranch to the loop. AddjudgmentShape(observation, { treeWithheld })next toobservationShapeinobservation.tsreturning what the judge is actually shown, so the loop compares one string and the vision-only rule lives next to the prompt rule.
Tests, two mutations survive:
- Deleting the location guard line in
JUDGMENT_REQUEST(prompts.ts:188) fails nothing. Pin it inagent-prompts.test.ts. - Dropping the 256-char cap for non-http locations (
observation.ts:314) fails nothing. Add a case like'com.example / ' + 'x'.repeat(300).
Nits:
EXTRACT_REQUESThas no location rule, but the changeset says extract sees the location. Say in the prompt whether extract may read values from the URL.config.mdx:957"...current screen with its location only." drop the trailing "only".agent.test.ts: the module-levelfixtureset inbeforeAlland mutated inside the fake model'srespond()is hidden coupling; move the release out ofrespond()or note it.
Fork CI (spec, mobile, docs, benchmark) is waiting on maintainer approval, so it hasn't run yet.
…he judge is shown prepareObservation now stores the location a judgment is shown, so buildPrompt reads it and no longer takes the app origin, and waitFor compares one string, judgmentShape, which leaves the location out under vision: 'only'. Extraction is told what the location is evidence for, and the tests pin both prompt lines and the cut of a long device location. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
[claude-opus-5-5] responding on behalf of two-pack Thanks for the review. All of it is in 9ff2dac.
Verified locally on 9ff2dac:
|
|
Automated review (Claude, on behalf of @okwasniewski). Re-reviewed 9ff2dac. All asks from the last review are done:
Optional nits, not blocking:
Fork CI still needs maintainer approval to run. Labeling Ready for Human Review. |
|
thanks! |
What and why
agent.assertandagent.waitForabout the address endedASSERTION_INCONCLUSIVEorSTEP_TIMEOUT, whether or not it matched.agent.actalready reads the location.buildPromptnow puts it on<observation>(packages/e2e/src/agent/prompts.ts:101). It is the same redacted valueactreads, a path on the app's origin and the whole address off it, without userinfo, cut at 256 characters like a link target (judgedLocationinobservation.ts).JUDGMENT_REQUESTgets one line: the location is evidence only for an instruction about the address, never for what the screen shows.waitForjudges again when only the location changes, as afterhistory.replaceState(index.ts:390). It skips that undervision: 'only', which is not shown the location.Issue
Fixes #1514
Verified
Ran it locally: yes
npm testandgpt-6-luna:toHaveURL('/page')The URL path is /pageASSERTION_INCONCLUSIVEThe URL path is /otherASSERTION_INCONCLUSIVEASSERTION_FAILEDas expectedThe URL path is /pageSTEP_TIMEOUTSTEP_TIMEOUTtests-agent,--strict-cachewithgpt-6-lunain place of the gateway model: 33 passed, noREPLAY_STALE.pnpm bench:abon the web benchmark: every test ran the same steps with the same outcome on both builds, and suite time wassame.--no-cache --ai-traceon the agent suites.Checklist
pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.!, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).docs/**/*.mdxpage for the behavior, andskills/e2e/if the skill describes it..d.tsreviewed,tests/types/sdk-types.tsupdated; wire change edits the schema and both fixtures.e2e/engine): changesets fore2e,@e2e-dev/web, and@e2e-dev/mobile.🤖 Generated with Claude Code