Skip to content

feat(geobenchx): add the GeoBenchX multistep GIS benchmark - #4649

Open
patdevinwilson wants to merge 6 commits into
NVIDIA-NeMo:mainfrom
patdevinwilson:pwilson/geobenchx-gym
Open

patdevinwilson wants to merge 6 commits into
NVIDIA-NeMo:mainfrom
patdevinwilson:pwilson/geobenchx-gym

Conversation

@patdevinwilson

@patdevinwilson patdevinwilson commented Oct 15, 2026 •

Copy link
Copy Markdown

** Legal has approved the GeoBenchX task data for evaluation and training;
redistribution in this repo is still under review. Until then this PR commits no task
data: task_data.py downloads it from the pinned upstream commit and checks its
SHA-256 (see DATA-ATTESTATION.md).

What does this PR do?

Adds a geobenchx resources server for the GeoBenchX multistep GIS agent benchmark
(202 tasks; Krechetova & Kochedykov 2025, arXiv:2503.18129), scored against the official
reference solutions, for Nemotron post-training evaluation and GRPO.

  • Scoring: compares the model's tool-call sequence with the upstream reference
    solutions. Unsolvable tasks that end with reject_task get full credit, as the
    paper's judge gives them.
  • Two tool backends:
    • Simulator (default): deterministic catalog and dataflow state. Needs no external
      data; the tests use it.
    • Real tools (configs/geobenchx_real_tools.yaml): the pinned upstream tool
      implementations, run on a local copy of the upstream data snapshot. They run in a
      pool of worker processes with per-session state, per-worker scratch directories,
      sanitized output paths and timeouts, plus fixes for two upstream tool defects.
  • Data: task records are downloaded from upstream commit 0edf6109, not committed.
    The fixed train/validation split (149 / 53) keeps near-duplicate task families
    (template variants, and unsolvable edits of solvable tasks) on one side, stratified by
    group and task kind. The prompt date is pinned, so the materialized JSONL rebuilds
    byte-identical.
  • Local Nemotron 3.5 Super VL: configs/super_vl_local.yaml plus vLLM serve and
    download scripts. Agent turns are capped at 25, the paper's limit.

The upstream data snapshot (about 507 MB, mixed source terms) is not redistributed.
Users download it themselves to use the real-tools backend.

Validation

  • python -m resources_servers.geobenchx.task_data: downloads the tasks, verifies the
    SHA-256, and writes benchmark / train / validation / example JSONL. These are
    byte-identical to the files built from the previously vendored copy.
  • pytest resources_servers/geobenchx/tests: 17 passed.
  • pre-commit run --files $(git ls-files resources_servers/geobenchx): all hooks pass.
  • gym env validate "+config_paths=[resources_servers/geobenchx/configs/geobenchx.yaml]":
    valid. The only warning is the expected one: no model server is defined.
  • Replaying all 222 upstream reference solutions through the real tools: 201 run with no
    tool error. The 21 failures are mistakes in the hand-written references (listed in the
    README).

Rollouts

Full real-tools runs (202 tasks × 4 repeats, 808 rollouts each) on three internal
Nemotron 3.5 Super VL checkpoints completed with no rollout failures. Scores and rollouts
are not attached: they are for unreleased checkpoints, and real-tools rollouts contain
data derived from the non-redistributable snapshot.

Compatibility and benchmark impact

None. This is a new resources server and changes no existing code. Simulator scores and
real-tools scores are not comparable; the README says so.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification.
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header.
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

Pre-commit was run on the changed files rather than with --all-files.

Give Nemotron post-training a reproducible Gym for the 202 GeoBenchX agent
tasks, scored against the official reference solutions.

- Score tool-call sequences against the upstream reference solutions, giving
  full credit to unsolvable tasks that end with reject_task, as the paper's
  judge does.
- Ship two tool backends: a deterministic catalog/dataflow simulator (default,
  no external data) and the pinned upstream GIS tools running on a locally
  mounted copy of the upstream data snapshot, in a pool of worker processes
  with per-session state, per-worker scratch directories, sanitized output
  paths, timeouts and fixes for two upstream tool defects.
- Download the task records from the pinned upstream commit and check their
  SHA-256 instead of committing them; build fixed, stratified 148/54
  train/validation splits with a pinned prompt date so rebuilds are
  byte-identical.
- Cap agent turns at 25 and add a config plus vLLM serve and download scripts
  for local Nemotron 3.5 Super VL endpoints.

Signed-off-by: patdevinwilson <pwilson@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@patdevinwilson

patdevinwilson commented Oct 19, 2026 •

Copy link
Copy Markdown
Author

@NVIDIA-NeMo/gym_maintainers could a vetter run /ok to test d7b802e70ddeedb981f006432ce7851492afe2f7 so CI runs on this draft? It stays a draft until legal finishes reviewing redistribution of the GeoBenchX task data (evaluation and training are approved); the task records are downloaded at setup rather than committed (see resources_servers/geobenchx/DATA-ATTESTATION.md). Review feedback on the resources server is welcome in the meantime.

Evaluation and training are approved; redistribution is still under review, so the task records stay uncommitted.

Signed-off-by: patdevinwilson <pwilson@nvidia.com>
@patdevinwilson
patdevinwilson marked this pull request as ready for review October 19, 2026 13:27
… split

16 of the 54 validation tasks had a near-duplicate in train: template variants with the same datasets and tool chain, or unsolvable edits of a solvable task. Group tasks into families and assign whole families, stratified by group and task kind with every group and kind in validation. The split becomes 149 train / 53 validation.

Signed-off-by: patdevinwilson <pwilson@nvidia.com>
@yaoyu-33 yaoyu-33 added area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer and removed community-request Issue reported or requested by someone from the community labels Oct 19, 2026
@github-actions github-actions Bot added the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 20, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond benchmark-onboarding Self-contained benchmark onboarding labels Oct 21, 2026
@cmunley1

Copy link
Copy Markdown
Contributor

/ok to test d7b802e

@cmunley1

Copy link
Copy Markdown
Contributor

/review

@nemo-automation-bot nemo-automation-bot Bot added the community-request Issue reported or requested by someone from the community label Oct 30, 2026
@nemo-automation-bot
nemo-automation-bot Bot requested review from a team October 30, 2026 21:06
@github-actions

Copy link
Copy Markdown
Contributor

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

The scorer can award full reward to invalid trajectories, and worker cancellation can misroute tool replies between rollouts. Additional RISK findings cover startup bounds, session retention, dataset/schema integration, metadata preservation, and verifier coverage.

Reviewed all 21 changed files and their directly affected callers, contracts, and tests across three independent topics. This was static inspection: no tests, model runs, downloads, or serving commands were executed. The downloaded official task records, upstream GIS implementation, and private model artifacts were unavailable; scoring counterexamples below use minimal synthetic references in the supported schema.

NOTE — Add published benchmark documentation

GeoBenchX has a component README, but no entry in the published environment guide. Please add a concise entry linking public preparation/run instructions. Preserve the simulator-versus-real-tools score warning and snapshot redistribution restrictions. Static acceptance: the published guide reaches a runnable public example without requiring private checkpoint access. This is a documentation improvement, not a demonstrated scoring failure.

NOTE — Include GeoBenchX in the generated README catalog

The root README's G entries omit the new server. The current metadata extraction reads dataset types only from agent blocks, so the new resources-owned declarations are skipped. Please add a narrow resources-owned-dataset fallback and regenerate the table through the existing hook, rather than hand-editing a row or moving dataset ownership. Preserve agent-owned entries and helper-config exclusion. Proposed acceptance: GeoBenchX appears with its train/validation metadata, existing entries remain intact, and a second generation produces no diff.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

Comment thread resources_servers/geobenchx/scoring.py Outdated
if "reject_task" in candidate_names:
return 0, {"reference_tools": reference_names, "matched_steps": 0}

unmatched = set(range(len(calls)))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Full-match scoring discards dependency order and dataflow.

Failure: For a synthetic reference load_geodata(Countries -> a); visualize_geographies([a]), reversing those calls or visualizing a nonexistent handle still matches every step and returns reward 1.0. This loop treats calls as an unordered bag, while _step_match skips all producer/consumer handles. Temporary-name invariance should preserve connections, not erase them. This affects multistep evaluation and training rewards.

Correction: Could _score_solution/_step_match maintain consistent reference-to-candidate bindings and producer-before-consumer dependencies? Independent loads may commute, but consumers must use the corresponding produced objects. Update the output-only rename test: it currently changes outputs without changing their consumers and therefore protects broken dataflow. An exact patch is uncertain because the documented malformed upstream references need an explicit compatibility policy rather than globally ignoring handles.

Preservation / proposed acceptance: Exact replay and consistent a -> b renaming must score 2; reversed dependent calls, missing producer/consumer handles, and visualization of an unrelated handle must not. Swapping independent loads, adding inspections, and appending steps after a complete solution within the call limit must remain valid. Preserve alternative-reference and reject/control behavior. Retain full credit for well-formed official references and explicitly account for known malformed references when updating the replay oracle. These checks are proposed, not executed.

try:
return _norm(json.loads(stripped))
except (json.JSONDecodeError, TypeError):
return " ".join(stripped.lower().split())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Normalization grants full reward to catalog names execution rejects.

Failure: Against a one-step load_geodata reference using Countries, a candidate using countries still scores 2 because _norm lowercases both. The simulator lookup is case-sensitive and returns unknown vector dataset. Thus an unsuccessful call receives full algorithmic reward independently of the dataflow issue.

Correction: Preserve exact catalog identifiers before generic normalization. In _step_match, immediately after actual = candidate_args.get(key), this focused addition would fix the confirmed catalog case:

        if key in {"dataset", "geodataset", "rasterdataset"}:
            if actual != expected:
                return False, partial
            continue

Please verify the suggestion before adopting it. Do not remove all normalization indiscriminately: serialized filter objects still need equivalence.

Preservation / proposed acceptance: Exact Countries must remain 2; lowercase countries must not be 2, and execute_tool must return its catalog error. Repeat for statistical/raster identifiers, including exact catalog whitespace. Preserve object-versus-JSON-string filter equivalence, numeric tolerance, consistent handle renaming, and the existing colormap-difference assertion of 1. No tests were executed.

proc.stdin.write((json.dumps(payload) + "\n").encode())
await proc.stdin.drain()
try:
line = await asyncio.wait_for(proc.stdout.readline(), timeout)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Cancellation leaves an outstanding reply on the shared worker stream.

Failure: Start a slow call for A, cancel it after dispatch, then issue a call for B routed to the same worker. Cancellation bypasses the TimeoutError handler and releases _lock, but the subprocess continues. B consumes A's reply, shifting subsequent replies too. This is reachable through ordinary disconnects: server startup installs cancellation middleware, which cancels in-flight handlers. Separate rollouts can receive each other's tool feedback.

Correction: Could request() retain transaction ownership across cancellation, covering startup, write/drain, and reply consumption? Either finish/drain the bounded transaction before releasing the lock, or terminate/reap the worker and explicitly invalidate its sessions. Shielding a detached read while releasing the lock is insufficient. The invariant is one matched reply per request with no outstanding reply when the next transaction starts. The exact patch depends on the chosen lost-session policy.

Preservation / proposed acceptance: With one worker, use a fake slow tool returning a unique marker. Cancel A after dispatch; B and two subsequent calls must receive their own markers. Cover cancellation during startup and same-session calls. Draining must preserve loaded state; restarting must report state loss explicitly. Retain the state-isolation, error-return, and timeout-restart assertions. Proposed checks only.

env=self._env,
limit=STREAM_LIMIT,
)
ready = json.loads(await self._proc.stdout.readline() or b"{}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Worker startup is outside the configured timeout.

Failure: Add time.sleep(60) at module import to the existing fake upstream and set tool_timeout_s=1. The first seed/tool request waits for readiness here before entering the timed response read, so its configured deadline is ineffective. An import that never completes holds the worker lock indefinitely; close() waits on that same lock. Session seeding immediately starts this handshake.

Correction: Give the readiness handshake a finite deadline and terminate/reap a partially started worker on failure before reuse. Bound startup and write/drain as well as the final response read, coordinating with cancellation cleanup. Distinguish startup failure from tool execution timeout. The exact timeout/cleanup implementation depends on that transaction policy; neither startup nor cleanup may wait indefinitely.

Preservation / proposed acceptance: A fake import exceeding the deadline must produce a specific bounded startup-timeout outcome, leave no live subprocess, and allow pool.close() to finish. Replacing it with healthy code must let a subsequent request start a fresh worker successfully. Preserve healthy initialization, persistent state, and the existing tool-timeout restart assertions. No tests were executed.

if isinstance(arguments, str):
try:
arguments = json.loads(arguments)
except json.JSONDecodeError:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Malformed rejection calls become valid, fully rewarded rejections.

Failure: On a rejection task, a function_call named reject_task with arguments "{" becomes ('reject_task', {}); the terminal-name branch then awards 2. But SimpleAgent does not execute calls whose JSON decoding fails. Non-object arguments such as "[]" are also converted into an apparently valid empty-object call. This rewards a rejection that never successfully occurred.

Correction: Preserve argument validity in the parser/scorer contract and require valid object arguments before a call can match, including zero-argument reject_task. Do not simply drop invalid calls: that would incorrectly turn attempted tool use into a zero-tool full match on control tasks. Keep attempts visible to terminal-order checks and step counts. The exact representation needs a coordinated parser/scorer change rather than an invented local sentinel patch.

Preservation / proposed acceptance: Through parse_calls plus score_task, valid reject_task arguments "{}" must score 2; "{" and "[]" must score 0. Valid inspection followed by valid rejection remains 2; valid rejection followed by another malformed attempted call remains 0. A no-step control reference must score 0, not 2, when there was malformed attempted tool use. Retain the existing reject/control assertions. Proposed checks only.

digest = hashlib.sha256(content).hexdigest()
if digest != UPSTREAM_TASKS_SHA256:
raise RuntimeError(f"{UPSTREAM_TASKS_URL} has sha256 {digest}, expected {UPSTREAM_TASKS_SHA256}")
partial = path.with_suffix(".partial")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Concurrent first-use downloads share a staging filename.

Failure: Two processes importing the module with an empty cache can both finish downloading before publication. Both write the same .partial path; after one replaces it, the other's partial.replace(path) can raise FileNotFoundError. Initialization fails despite both downloads passing checksum verification.

Correction: Give each writer private staging. With import tempfile added, a small replacement for lines 75–77 is:

    with tempfile.TemporaryDirectory(dir=path.parent, prefix=f".{path.name}.") as temporary:
        partial = Path(temporary) / path.name
        partial.write_bytes(content)
        partial.replace(path)

Please verify the suggestion before adopting it, and retain it if preparation moves out of this module. Successful writers can atomically publish identical verified bytes without removing each other's staging file.

Preservation / proposed acceptance: Synchronize two fetches so both verified downloads finish before publication; both must return the destination, whose SHA-256 must match the pin. A mismatched download must leave an existing cache unchanged, staging must be cleaned up, and subsequent cached loading must work offline. Preserve checksum verification before publication. No tests were executed.

VERIFIER_FIXTURE = VerifierFixture(
server_factory=GeoBenchXVerifier,
request_model=GeoBenchXVerifyRequest,
cases_path=Path(__file__).parent / "tests" / "verifier_cases.jsonl",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — The exported verifier fixture points to an absent file.

Failure: The immutable tests directory contains only the two Python files, not this JSONL. Calling exercise_verifier_fixture(VERIFIER_FIXTURE, reward_range=(0, 1), determinism='unknown') therefore fails before exercising verification. Fixture loading explicitly rejects a missing file. The current component tests call the scorer rather than the exported request/verifier contract.

Correction: Supply a runnable fixture with full-reward, zero-reward, and malformed-request cases, and exercise it in component tests. Respect redistribution restrictions: independently authored synthetic cases and a fixture-only synthetic task lookup can cover the contract without committing official records. Keep the production lookup unchanged. Removing the declaration alone would leave the new verifier request handling untested; the fixture wiring needs a deliberate implementation.

Preservation / proposed acceptance: Execute the fixture through GeoBenchXVerifyRequest and verification, asserting rewards exactly 1.0 and 0.0; a request missing task_id must fail validation naming that field. It must require neither GIS dependencies nor a model server. Retain official-reference, reject/control, and colormap-partial scorer assertions. These are proposed checks, not executed results.

if state is None:
state = self._sessions[session] = _new_state()
self._sessions.move_to_end(session)
while len(self._sessions) > self._max_sessions:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — The LRU limit silently evicts live rollout state.

Failure: With one worker and max_sessions_per_worker=1, A loads gdp; seed/use B; then A calls list_handles. This loop evicts A, and its next call silently constructs empty state rather than returning gdp. Both rollouts may still be active: the worker protocol has no release operation, and the configured legacy run path does not release worker sessions. More concurrent sessions hashing to one worker trigger the same problem at the default limit.

Correction: Distinguish active sessions from reclaimable completed sessions. Wire release through the applicable lifecycle, reclaim only released state, and reject/queue admission when capacity belongs to live sessions. Lost sessions must not silently resume as new valid sessions. This requires coordinated worker/pool/server lifecycle changes; simply removing eviction would cause unbounded retention, so no certain local patch is proposed.

Preservation / proposed acceptance: With capacity one, B must wait or fail explicitly while A retains gdp, or be admitted after A is released. After release, B must succeed and A must not silently resume empty. Preserve intentional reset clearing only its own state and isolation between admitted sessions, including the existing concrete assertions. No tests were executed.


| Variant | Checkpoint | Role |
| --- | --- | --- |
| `bf16` | `Nemotron-3.5-Super-VL-120B-A12B-GA-candidate-MTP-boosted-20261001` | Training start |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Remove private checkpoint details and employee-specific paths from public examples.

The candidate table and run plan identify unreleased checkpoints. The download script explicitly describes its repositories as private, and it and the serve script embed an employee-specific absolute path. This conflicts with the protected public-repository content guidance, independently of whether outsiders can download the artifacts. This is not a claim of credential disclosure.

Correction: Please replace these with public examples or caller-supplied repository/path variables, move restricted run instructions to approved private systems, and use a portable directory default. Update the README, scripts, and related config comments consistently.

Preservation / acceptance: Retain the supported serving layouts, target-versus-draft distinction, matching download/serve directories, and GPU-conflict checks. Static acceptance: no private checkpoint identifiers or employee-specific absolute paths remain in public examples. Proposed stubbed-shell checks should show supplied repositories download to the directories serving mounts and the draft variant mounts both target and draft. These checks were not executed.

description: GeoBenchX 202-task multistep geospatial tool-calling regression benchmark.
datasets:
- name: benchmark
type: example

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — The full benchmark is declared as an example, so standard benchmark execution cannot select it.

Failure: Even after manual materialization, gym eval run with this config and --split benchmark fails because no benchmark-type dataset is declared. The E2E runner validates declared split types before dispatch. gym eval prepare also finds nothing preparable: all entries lack prepare_script, and discovery only selects benchmark entries or explicit preparers. The documented no-serve file workflow avoids, but does not repair, this integration gap.

Correction: Declare this entry as type: benchmark with a prepare_script. After separating preparation from the reserved schema module, expose a prepare() wrapper that materializes the records and returns the exact configured benchmark path; Gym verifies that returned-path contract. Keep train/validation/example entries in their respective types. The wrapper body can be:

def prepare() -> Path:
    materialize()
    return DATASET_PATH

Please verify it in the final preparation module and wire that module's path into the YAML.

Preservation / proposed acceptance: Preparation discovery must find the benchmark; prepare() must return the configured 202-row file; --split benchmark must pass split selection and collate those rows. Preserve 149/53 partitioning, the five-row example, no-serve input usage, and the prohibition on committing official records. No commands were executed.

@github-actions github-actions Bot removed the sla:triage-overdue Review assignment is over the one-business-day SLA label Oct 30, 2026
@svcnvidia-nemo-ci svcnvidia-nemo-ci added waiting-on-maintainers Waiting on maintainers to respond and removed waiting-on-maintainers Waiting on maintainers to respond labels Oct 31, 2026
Signed-off-by: patdevinwilson <pwilson@nvidia.com>
…ring, worker lifecycle

- prepare_data keeps the original split, prompt, and row logic (byte-identical
  output), adds unique staging and prepare() for gym eval prepare.
- The scorer enforces producer-before-consumer order, fixes the inverted
  exact-match flag, skips upstream-local raster paths, caps malformed references
  at partial credit, and bounds search time.
- Workers: request ids, startup timeout, boot-time start, cancellation that
  keeps the worker, release on verify/close/retire, idle reclaim, explicit
  lost-state errors, and per-request error replies.

Signed-off-by: patdevinwilson <pwilson@nvidia.com>
Signed-off-by: patdevinwilson <pwilson@nvidia.com>

# Conflicts:
#	README.md

This branch was previously deployed

1 inactive (outdated) deployment
public — d7b802e7 Deployed Oct 30, 2026 by copy-pr-bot[bot] via release / finalize / notify #8855
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:environment Individual environments, benchmarks, verifiers, and environment-specific resources servers benchmark-onboarding Self-contained benchmark onboarding community-request Issue reported or requested by someone from the community complexity:high Cross-component or shared-contract change with a large review surface feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants