Skip to content

fix(agent): show the judge the current location - #1515

Merged
okwasniewski merged 3 commits into
e2e-inc:mainfrom
two-pack:fix/judge-current-url
Nov 9, 2026
Merged

okwasniewski merged 3 commits into
e2e-inc:mainfrom
two-pack:fix/judge-current-url

Conversation

@two-pack

@two-pack two-pack commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

What and why

  • The built-in judge never saw the current URL, so agent.assert and agent.waitFor about the address ended ASSERTION_INCONCLUSIVE or STEP_TIMEOUT, whether or not it matched. agent.act already reads the location.
  • buildPrompt now puts it on <observation> (packages/e2e/src/agent/prompts.ts:101). It is the same redacted value act reads, a path on the app's origin and the whole address off it, without userinfo, cut at 256 characters like a link target (judgedLocation in observation.ts).
  • JUDGMENT_REQUEST gets one line: the location is evidence only for an instruction about the address, never for what the screen shows.
  • waitFor judges again when only the location changes, as after history.replaceState (index.ts:390). It skips that under vision: 'only', which is not shown the location.
<observation revision="b1" location="/page" viewport="1280x720">

Issue

Fixes #1514

Verified

Ran it locally: yes

Test main this branch
control, toHaveURL('/page') passes passes
assert A, The URL path is /page ASSERTION_INCONCLUSIVE passes
assert B, The URL path is /other ASSERTION_INCONCLUSIVE ASSERTION_FAILED as expected
waitFor A, The URL path is /page STEP_TIMEOUT passes
waitFor B, only the URL changes STEP_TIMEOUT passes
  • Web benchmark tests-agent, --strict-cache with gpt-6-luna in place of the gateway model: 33 passed, no REPLAY_STALE.
  • pnpm bench:ab on the web benchmark: every test ran the same steps with the same outcome on both builds, and suite time was same.
  • Not run: --no-cache --ai-trace on the agent suites.

Checklist

  • Changeset added (pnpm changeset) for any change to a published package, or the change is testbed, docs, or CI only.
  • Breaking change: title carries !, the changeset body names what breaks and what replaces it, and the deprecation warning landed a window earlier (stability policy in CONTRIBUTING.md).
  • Docs updated in the same PR: the docs/**/*.mdx page for the behavior, and skills/e2e/ if the skill describes it.
  • Contract change: emitted .d.ts reviewed, tests/types/sdk-types.ts updated; wire change edits the schema and both fixtures.
  • Engine contract change (e2e/engine): changesets for e2e, @e2e-dev/web, and @e2e-dev/mobile.

🤖 Generated with Claude Code

View guided diff

The built-in judge never saw the current URL, so an assertion or wait about
the address ended inconclusive or timed out whatever the address was.
buildPrompt now puts the redacted location on <observation> as a
`location` attribute: a path on the app's origin, the whole address off it,
without userinfo, cut like a link target. waitFor also judges again when only
the location changes, except under vision: 'only', which is not shown it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the size:L Up to 499 changed lines, not counting tests, fixtures, translated docs, or the lockfile label Nov 3, 2026

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 18 files

Tip: cubic used a learning from your PR history. Let your coding agent read cubic learnings directly with the cubic MCP.

View guided diff | Re-trigger cubic

Comment thread docs/reference/agent.mdx Resolved
Comment thread packages/e2e/src/agent/index.ts Resolved
Comment thread docs/core-concepts.mdx Resolved
Comment thread docs/core-concepts.mdx Resolved
Comment thread docs/reference/security.mdx Resolved
Comment thread docs/reference/config.mdx Resolved
Comment thread docs/reference/security.mdx Resolved
Comment thread packages/e2e/src/agent/index.ts Resolved
Comment thread packages/e2e/src/agent/invocation.ts Resolved
…nly' skips

A wait ignored a clock-shaped value only to keep a device's screen title
from counting as a change; it also hid a real move such as /at/10:15 to
/at/10:16. Only an http(s) address is now compared as it is, so a bare
device title such as "Timer: 00:12" still ignores its clock. The docs and
changeset say a location change alone does not wake a vision-only wait,
the security page says an off-origin address reaches the judge whole, and
the lines that list what a judgment sees name its params and the agent's
context too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@okwasniewski okwasniewski left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review (agent-run thermo-nuclear review, a human maintainer reviews before merge).

Thanks, this is a real bug and a clean fix. Checked at 6f76a74: typecheck, lint, unit (3262) and integration agent/agent-vision/agent-policy pass. Redaction matches act (same observation.location, prompt redacted again by Invocation), no cache-key impact (judgments are never cached, act keys don't include prompts), mobile location (bundleId / title) behaves sanely. Your declines on the cubic threads hold up.

Two structural changes before this is ready:

  1. Compute the judge's location in prepareObservation, not buildPrompt. prepareObservation already gets appOrigin (invocation.ts:244, observation-feed.ts:403). Store the judged location as its own field on AgentObservation (not path, the replay cache's sameRoute uses that). This deletes PromptInput.appOrigin (prompts.ts:63-64), the appOrigin line in invocation.ts:335 and the judgedLocation call in buildPrompt (prompts.ts:101), and lets locationShape read the stored field instead of re-running judgedLocation(..., undefined) and re-parsing its own output with isHttpUrl (observation.ts:326-329).
  2. One comparison in waitForNextJudgment. index.ts:357,389-390 adds a second shape plus a treeWithheld branch to the loop. Add judgmentShape(observation, { treeWithheld }) next to observationShape in observation.ts returning what the judge is actually shown, so the loop compares one string and the vision-only rule lives next to the prompt rule.

Tests, two mutations survive:

  • Deleting the location guard line in JUDGMENT_REQUEST (prompts.ts:188) fails nothing. Pin it in agent-prompts.test.ts.
  • Dropping the 256-char cap for non-http locations (observation.ts:314) fails nothing. Add a case like 'com.example / ' + 'x'.repeat(300).

Nits:

  • EXTRACT_REQUEST has no location rule, but the changeset says extract sees the location. Say in the prompt whether extract may read values from the URL.
  • config.mdx:957 "...current screen with its location only." drop the trailing "only".
  • agent.test.ts: the module-level fixture set in beforeAll and mutated inside the fake model's respond() is hidden coupling; move the release out of respond() or note it.

Fork CI (spec, mobile, docs, benchmark) is waiting on maintainer approval, so it hasn't run yet.

…he judge is shown

prepareObservation now stores the location a judgment is shown, so
buildPrompt reads it and no longer takes the app origin, and waitFor
compares one string, judgmentShape, which leaves the location out under
vision: 'only'. Extraction is told what the location is evidence for, and
the tests pin both prompt lines and the cut of a long device location.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@two-pack

two-pack commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

[claude-opus-5-5] responding on behalf of two-pack

Thanks for the review. All of it is in 9ff2dac.

  1. prepareObservation now stores the judge's location as its own field, judgedLocation (observation.ts), next to path, which stays as the replay cache reads it. PromptInput.appOrigin, the appOrigin line in invocation.ts, and the judgedLocation call in buildPrompt are gone; buildPrompt reads the field.
  2. waitForNextJudgment compares one string, judgmentShape(observation, { treeWithheld }), added next to observationShape. It leaves the location out under vision: 'only', and keeps the clock rule for a device title without re-parsing. One small difference from a pure refactor: the wait now compares the location as the judge is shown it, so on the app's origin a change near the end of a path longer than the 256-character cut now counts as a change, as the judge would see it.
  3. Both mutations now fail a test in agent-prompts.test.ts: one pins the location line of JUDGMENT_REQUEST, one cuts a com.example / + 300 characters location. I ran each mutation to confirm.
  4. EXTRACT_REQUEST now says a value may come from the location only when the instruction asks about the address or route.
  5. config.mdx: dropped the trailing "only".
  6. agent.test.ts: kept the release inside respond(), since a release from the test would race the first judgment, and documented that on fixture.

Verified locally on 9ff2dac:

  • pnpm check, and packages/e2e tests (4108 passed). Each of the two mutations you listed now fails a test; I applied both to confirm.
  • The issue's reproduction with gpt-6-luna: 5 of 5 pass.
  • The web benchmark's tests-agent with --strict-cache and gpt-6-luna in place of the gateway model: 33 passed.
  • One-off checks with gpt-6-luna, not committed as tests:
    • agent.extract read status=active from the address when asked for it, and read the heading from the screen when the query held another value.
    • A query that disagrees with the screen did not fail an agent.assert about the screen.
    • On /settings with no settings form, agent.assert('The settings form is shown') did not pass on the path alone.
  • Not run: the agent suites with --no-cache --ai-trace.

@okwasniewski

Copy link
Copy Markdown
Member

Automated review (Claude, on behalf of @okwasniewski).

Re-reviewed 9ff2dac. All asks from the last review are done:

  • judgedLocation is its own AgentObservation field set in prepareObservation, PromptInput.appOrigin is gone, path untouched so replay sameRoute is unaffected.
  • One judgmentShape(observation, { treeWithheld }) used by waitForNextJudgment.
  • Both previously surviving mutations (JUDGMENT_REQUEST location line, non-http 256 cap) now fail tests. 7 more mutations on the change (userinfo, origin-relative path, treeWithheld, clock normalization, prompt attribute, shape location, EXTRACT_REQUEST line) all caught.
  • Lint, typecheck, unit (3265 pass), agent integration suites, docs/changeset checks pass locally. Merges clean with main.

Optional nits, not blocking:

  • packages/e2e/src/agent/index.ts:333-334 JSDoc reads garbled ("because the settles after an action wait on that").
  • index.ts:392 long line, a local shapeOf helper would cover 361 and 392.
  • JudgedLocation.url reads vague, isHttpUrl says what it is.
  • judgmentShape tests fit agent-observation.test.ts better than agent-prompts.test.ts.
  • AGENTS.md:554,561, config.mdx:957 edited lines not rewrapped.

Fork CI still needs maintainer approval to run. Labeling Ready for Human Review.

@okwasniewski
okwasniewski dismissed their stale review November 3, 2026 16:12

Addressed, see re-review comment.

@okwasniewski

Copy link
Copy Markdown
Member

thanks!

@okwasniewski
okwasniewski merged commit ed9e633 into e2e-inc:main Nov 9, 2026
25 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for Human Review size:L Up to 499 changed lines, not counting tests, fixtures, translated docs, or the lockfile

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent: assert and waitFor cannot judge the current URL

2 participants