Repository navigation
feat(geobenchx): add the GeoBenchX multistep GIS benchmark - #4649
patdevinwilson wants to merge 6 commits into
Conversation
Give Nemotron post-training a reproducible Gym for the 202 GeoBenchX agent tasks, scored against the official reference solutions. - Score tool-call sequences against the upstream reference solutions, giving full credit to unsolvable tasks that end with reject_task, as the paper's judge does. - Ship two tool backends: a deterministic catalog/dataflow simulator (default, no external data) and the pinned upstream GIS tools running on a locally mounted copy of the upstream data snapshot, in a pool of worker processes with per-session state, per-worker scratch directories, sanitized output paths, timeouts and fixes for two upstream tool defects. - Download the task records from the pinned upstream commit and check their SHA-256 instead of committing them; build fixed, stratified 148/54 train/validation splits with a pinned prompt date so rebuilds are byte-identical. - Cap agent turns at 25 and add a config plus vLLM serve and download scripts for local Nemotron 3.5 Super VL endpoints. Signed-off-by: patdevinwilson <pwilson@nvidia.com>
|
@NVIDIA-NeMo/gym_maintainers could a vetter run |
Evaluation and training are approved; redistribution is still under review, so the task records stay uncommitted. Signed-off-by: patdevinwilson <pwilson@nvidia.com>
… split 16 of the 54 validation tasks had a near-duplicate in train: template variants with the same datasets and tool chain, or unsolvable edits of a solvable task. Group tasks into families and assign whole families, stratified by group and task kind with every group and kind in validation. The split becomes 149 train / 53 validation. Signed-off-by: patdevinwilson <pwilson@nvidia.com>
|
/ok to test d7b802e |
|
/review |
|
🌿 Preview your docs: https://nvidia-preview-pr-4649-d7b802e70dde.docs.buildwithfern.com/nemo/gym |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
The scorer can award full reward to invalid trajectories, and worker cancellation can misroute tool replies between rollouts. Additional RISK findings cover startup bounds, session retention, dataset/schema integration, metadata preservation, and verifier coverage.
Reviewed all 21 changed files and their directly affected callers, contracts, and tests across three independent topics. This was static inspection: no tests, model runs, downloads, or serving commands were executed. The downloaded official task records, upstream GIS implementation, and private model artifacts were unavailable; scoring counterexamples below use minimal synthetic references in the supported schema.
NOTE — Add published benchmark documentation
GeoBenchX has a component README, but no entry in the published environment guide. Please add a concise entry linking public preparation/run instructions. Preserve the simulator-versus-real-tools score warning and snapshot redistribution restrictions. Static acceptance: the published guide reaches a runnable public example without requiring private checkpoint access. This is a documentation improvement, not a demonstrated scoring failure.
NOTE — Include GeoBenchX in the generated README catalog
The root README's G entries omit the new server. The current metadata extraction reads dataset types only from agent blocks, so the new resources-owned declarations are skipped. Please add a narrow resources-owned-dataset fallback and regenerate the table through the existing hook, rather than hand-editing a row or moving dataset ownership. Preserve agent-owned entries and helper-config exclusion. Proposed acceptance: GeoBenchX appears with its train/validation metadata, existing entries remain intact, and a second generation produces no diff.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| if "reject_task" in candidate_names: | ||
| return 0, {"reference_tools": reference_names, "matched_steps": 0} | ||
|
|
||
| unmatched = set(range(len(calls))) |
There was a problem hiding this comment.
BLOCKER — Full-match scoring discards dependency order and dataflow.
Failure: For a synthetic reference load_geodata(Countries -> a); visualize_geographies([a]), reversing those calls or visualizing a nonexistent handle still matches every step and returns reward 1.0. This loop treats calls as an unordered bag, while _step_match skips all producer/consumer handles. Temporary-name invariance should preserve connections, not erase them. This affects multistep evaluation and training rewards.
Correction: Could _score_solution/_step_match maintain consistent reference-to-candidate bindings and producer-before-consumer dependencies? Independent loads may commute, but consumers must use the corresponding produced objects. Update the output-only rename test: it currently changes outputs without changing their consumers and therefore protects broken dataflow. An exact patch is uncertain because the documented malformed upstream references need an explicit compatibility policy rather than globally ignoring handles.
Preservation / proposed acceptance: Exact replay and consistent a -> b renaming must score 2; reversed dependent calls, missing producer/consumer handles, and visualization of an unrelated handle must not. Swapping independent loads, adding inspections, and appending steps after a complete solution within the call limit must remain valid. Preserve alternative-reference and reject/control behavior. Retain full credit for well-formed official references and explicitly account for known malformed references when updating the replay oracle. These checks are proposed, not executed.
| try: | ||
| return _norm(json.loads(stripped)) | ||
| except (json.JSONDecodeError, TypeError): | ||
| return " ".join(stripped.lower().split()) |
There was a problem hiding this comment.
BLOCKER — Normalization grants full reward to catalog names execution rejects.
Failure: Against a one-step load_geodata reference using Countries, a candidate using countries still scores 2 because _norm lowercases both. The simulator lookup is case-sensitive and returns unknown vector dataset. Thus an unsuccessful call receives full algorithmic reward independently of the dataflow issue.
Correction: Preserve exact catalog identifiers before generic normalization. In _step_match, immediately after actual = candidate_args.get(key), this focused addition would fix the confirmed catalog case:
if key in {"dataset", "geodataset", "rasterdataset"}:
if actual != expected:
return False, partial
continuePlease verify the suggestion before adopting it. Do not remove all normalization indiscriminately: serialized filter objects still need equivalence.
Preservation / proposed acceptance: Exact Countries must remain 2; lowercase countries must not be 2, and execute_tool must return its catalog error. Repeat for statistical/raster identifiers, including exact catalog whitespace. Preserve object-versus-JSON-string filter equivalence, numeric tolerance, consistent handle renaming, and the existing colormap-difference assertion of 1. No tests were executed.
| proc.stdin.write((json.dumps(payload) + "\n").encode()) | ||
| await proc.stdin.drain() | ||
| try: | ||
| line = await asyncio.wait_for(proc.stdout.readline(), timeout) |
There was a problem hiding this comment.
BLOCKER — Cancellation leaves an outstanding reply on the shared worker stream.
Failure: Start a slow call for A, cancel it after dispatch, then issue a call for B routed to the same worker. Cancellation bypasses the TimeoutError handler and releases _lock, but the subprocess continues. B consumes A's reply, shifting subsequent replies too. This is reachable through ordinary disconnects: server startup installs cancellation middleware, which cancels in-flight handlers. Separate rollouts can receive each other's tool feedback.
Correction: Could request() retain transaction ownership across cancellation, covering startup, write/drain, and reply consumption? Either finish/drain the bounded transaction before releasing the lock, or terminate/reap the worker and explicitly invalidate its sessions. Shielding a detached read while releasing the lock is insufficient. The invariant is one matched reply per request with no outstanding reply when the next transaction starts. The exact patch depends on the chosen lost-session policy.
Preservation / proposed acceptance: With one worker, use a fake slow tool returning a unique marker. Cancel A after dispatch; B and two subsequent calls must receive their own markers. Cover cancellation during startup and same-session calls. Draining must preserve loaded state; restarting must report state loss explicitly. Retain the state-isolation, error-return, and timeout-restart assertions. Proposed checks only.
| env=self._env, | ||
| limit=STREAM_LIMIT, | ||
| ) | ||
| ready = json.loads(await self._proc.stdout.readline() or b"{}") |
There was a problem hiding this comment.
BLOCKER — Worker startup is outside the configured timeout.
Failure: Add time.sleep(60) at module import to the existing fake upstream and set tool_timeout_s=1. The first seed/tool request waits for readiness here before entering the timed response read, so its configured deadline is ineffective. An import that never completes holds the worker lock indefinitely; close() waits on that same lock. Session seeding immediately starts this handshake.
Correction: Give the readiness handshake a finite deadline and terminate/reap a partially started worker on failure before reuse. Bound startup and write/drain as well as the final response read, coordinating with cancellation cleanup. Distinguish startup failure from tool execution timeout. The exact timeout/cleanup implementation depends on that transaction policy; neither startup nor cleanup may wait indefinitely.
Preservation / proposed acceptance: A fake import exceeding the deadline must produce a specific bounded startup-timeout outcome, leave no live subprocess, and allow pool.close() to finish. Replacing it with healthy code must let a subsequent request start a fresh worker successfully. Preserve healthy initialization, persistent state, and the existing tool-timeout restart assertions. No tests were executed.
| if isinstance(arguments, str): | ||
| try: | ||
| arguments = json.loads(arguments) | ||
| except json.JSONDecodeError: |
There was a problem hiding this comment.
RISK — Malformed rejection calls become valid, fully rewarded rejections.
Failure: On a rejection task, a function_call named reject_task with arguments "{" becomes ('reject_task', {}); the terminal-name branch then awards 2. But SimpleAgent does not execute calls whose JSON decoding fails. Non-object arguments such as "[]" are also converted into an apparently valid empty-object call. This rewards a rejection that never successfully occurred.
Correction: Preserve argument validity in the parser/scorer contract and require valid object arguments before a call can match, including zero-argument reject_task. Do not simply drop invalid calls: that would incorrectly turn attempted tool use into a zero-tool full match on control tasks. Keep attempts visible to terminal-order checks and step counts. The exact representation needs a coordinated parser/scorer change rather than an invented local sentinel patch.
Preservation / proposed acceptance: Through parse_calls plus score_task, valid reject_task arguments "{}" must score 2; "{" and "[]" must score 0. Valid inspection followed by valid rejection remains 2; valid rejection followed by another malformed attempted call remains 0. A no-step control reference must score 0, not 2, when there was malformed attempted tool use. Retain the existing reject/control assertions. Proposed checks only.
| digest = hashlib.sha256(content).hexdigest() | ||
| if digest != UPSTREAM_TASKS_SHA256: | ||
| raise RuntimeError(f"{UPSTREAM_TASKS_URL} has sha256 {digest}, expected {UPSTREAM_TASKS_SHA256}") | ||
| partial = path.with_suffix(".partial") |
There was a problem hiding this comment.
RISK — Concurrent first-use downloads share a staging filename.
Failure: Two processes importing the module with an empty cache can both finish downloading before publication. Both write the same .partial path; after one replaces it, the other's partial.replace(path) can raise FileNotFoundError. Initialization fails despite both downloads passing checksum verification.
Correction: Give each writer private staging. With import tempfile added, a small replacement for lines 75–77 is:
with tempfile.TemporaryDirectory(dir=path.parent, prefix=f".{path.name}.") as temporary:
partial = Path(temporary) / path.name
partial.write_bytes(content)
partial.replace(path)Please verify the suggestion before adopting it, and retain it if preparation moves out of this module. Successful writers can atomically publish identical verified bytes without removing each other's staging file.
Preservation / proposed acceptance: Synchronize two fetches so both verified downloads finish before publication; both must return the destination, whose SHA-256 must match the pin. A mismatched download must leave an existing cache unchanged, staging must be cleaned up, and subsequent cached loading must work offline. Preserve checksum verification before publication. No tests were executed.
| VERIFIER_FIXTURE = VerifierFixture( | ||
| server_factory=GeoBenchXVerifier, | ||
| request_model=GeoBenchXVerifyRequest, | ||
| cases_path=Path(__file__).parent / "tests" / "verifier_cases.jsonl", |
There was a problem hiding this comment.
RISK — The exported verifier fixture points to an absent file.
Failure: The immutable tests directory contains only the two Python files, not this JSONL. Calling exercise_verifier_fixture(VERIFIER_FIXTURE, reward_range=(0, 1), determinism='unknown') therefore fails before exercising verification. Fixture loading explicitly rejects a missing file. The current component tests call the scorer rather than the exported request/verifier contract.
Correction: Supply a runnable fixture with full-reward, zero-reward, and malformed-request cases, and exercise it in component tests. Respect redistribution restrictions: independently authored synthetic cases and a fixture-only synthetic task lookup can cover the contract without committing official records. Keep the production lookup unchanged. Removing the declaration alone would leave the new verifier request handling untested; the fixture wiring needs a deliberate implementation.
Preservation / proposed acceptance: Execute the fixture through GeoBenchXVerifyRequest and verification, asserting rewards exactly 1.0 and 0.0; a request missing task_id must fail validation naming that field. It must require neither GIS dependencies nor a model server. Retain official-reference, reject/control, and colormap-partial scorer assertions. These are proposed checks, not executed results.
| if state is None: | ||
| state = self._sessions[session] = _new_state() | ||
| self._sessions.move_to_end(session) | ||
| while len(self._sessions) > self._max_sessions: |
There was a problem hiding this comment.
RISK — The LRU limit silently evicts live rollout state.
Failure: With one worker and max_sessions_per_worker=1, A loads gdp; seed/use B; then A calls list_handles. This loop evicts A, and its next call silently constructs empty state rather than returning gdp. Both rollouts may still be active: the worker protocol has no release operation, and the configured legacy run path does not release worker sessions. More concurrent sessions hashing to one worker trigger the same problem at the default limit.
Correction: Distinguish active sessions from reclaimable completed sessions. Wire release through the applicable lifecycle, reclaim only released state, and reject/queue admission when capacity belongs to live sessions. Lost sessions must not silently resume as new valid sessions. This requires coordinated worker/pool/server lifecycle changes; simply removing eviction would cause unbounded retention, so no certain local patch is proposed.
Preservation / proposed acceptance: With capacity one, B must wait or fail explicitly while A retains gdp, or be admitted after A is released. After release, B must succeed and A must not silently resume empty. Preserve intentional reset clearing only its own state and isolation between admitted sessions, including the existing concrete assertions. No tests were executed.
|
|
||
| | Variant | Checkpoint | Role | | ||
| | --- | --- | --- | | ||
| | `bf16` | `Nemotron-3.5-Super-VL-120B-A12B-GA-candidate-MTP-boosted-20261001` | Training start | |
There was a problem hiding this comment.
RISK — Remove private checkpoint details and employee-specific paths from public examples.
The candidate table and run plan identify unreleased checkpoints. The download script explicitly describes its repositories as private, and it and the serve script embed an employee-specific absolute path. This conflicts with the protected public-repository content guidance, independently of whether outsiders can download the artifacts. This is not a claim of credential disclosure.
Correction: Please replace these with public examples or caller-supplied repository/path variables, move restricted run instructions to approved private systems, and use a portable directory default. Update the README, scripts, and related config comments consistently.
Preservation / acceptance: Retain the supported serving layouts, target-versus-draft distinction, matching download/serve directories, and GPU-conflict checks. Static acceptance: no private checkpoint identifiers or employee-specific absolute paths remain in public examples. Proposed stubbed-shell checks should show supplied repositories download to the directories serving mounts and the draft variant mounts both target and draft. These checks were not executed.
| description: GeoBenchX 202-task multistep geospatial tool-calling regression benchmark. | ||
| datasets: | ||
| - name: benchmark | ||
| type: example |
There was a problem hiding this comment.
RISK — The full benchmark is declared as an example, so standard benchmark execution cannot select it.
Failure: Even after manual materialization, gym eval run with this config and --split benchmark fails because no benchmark-type dataset is declared. The E2E runner validates declared split types before dispatch. gym eval prepare also finds nothing preparable: all entries lack prepare_script, and discovery only selects benchmark entries or explicit preparers. The documented no-serve file workflow avoids, but does not repair, this integration gap.
Correction: Declare this entry as type: benchmark with a prepare_script. After separating preparation from the reserved schema module, expose a prepare() wrapper that materializes the records and returns the exact configured benchmark path; Gym verifies that returned-path contract. Keep train/validation/example entries in their respective types. The wrapper body can be:
def prepare() -> Path:
materialize()
return DATASET_PATHPlease verify it in the final preparation module and wire that module's path into the YAML.
Preservation / proposed acceptance: Preparation discovery must find the benchmark; prepare() must return the configured 202-row file; --split benchmark must pass split selection and collate those rows. Preserve 149/53 partitioning, the five-row example, no-serve input usage, and the prohibition on committing official records. No commands were executed.
Signed-off-by: patdevinwilson <pwilson@nvidia.com>
7612e49 to
276ece4
Compare
…ring, worker lifecycle - prepare_data keeps the original split, prompt, and row logic (byte-identical output), adds unique staging and prepare() for gym eval prepare. - The scorer enforces producer-before-consumer order, fixes the inverted exact-match flag, skips upstream-local raster paths, caps malformed references at partial credit, and bounds search time. - Workers: request ids, startup timeout, boot-time start, cancellation that keeps the worker, release on verify/close/retire, idle reclaim, explicit lost-state errors, and per-request error replies. Signed-off-by: patdevinwilson <pwilson@nvidia.com>
Signed-off-by: patdevinwilson <pwilson@nvidia.com> # Conflicts: # README.md
What does this PR do?
Adds a
geobenchxresources server for the GeoBenchX multistep GIS agent benchmark(202 tasks; Krechetova & Kochedykov 2025, arXiv:2503.18129), scored against the official
reference solutions, for Nemotron post-training evaluation and GRPO.
solutions. Unsolvable tasks that end with
reject_taskget full credit, as thepaper's judge gives them.
data; the tests use it.
configs/geobenchx_real_tools.yaml): the pinned upstream toolimplementations, run on a local copy of the upstream data snapshot. They run in a
pool of worker processes with per-session state, per-worker scratch directories,
sanitized output paths and timeouts, plus fixes for two upstream tool defects.
0edf6109, not committed.The fixed train/validation split (149 / 53) keeps near-duplicate task families
(template variants, and unsolvable edits of solvable tasks) on one side, stratified by
group and task kind. The prompt date is pinned, so the materialized JSONL rebuilds
byte-identical.
configs/super_vl_local.yamlplus vLLM serve anddownload scripts. Agent turns are capped at 25, the paper's limit.
The upstream data snapshot (about 507 MB, mixed source terms) is not redistributed.
Users download it themselves to use the real-tools backend.
Validation
python -m resources_servers.geobenchx.task_data: downloads the tasks, verifies theSHA-256, and writes
benchmark/train/validation/exampleJSONL. These arebyte-identical to the files built from the previously vendored copy.
pytest resources_servers/geobenchx/tests: 17 passed.pre-commit run --files $(git ls-files resources_servers/geobenchx): all hooks pass.gym env validate "+config_paths=[resources_servers/geobenchx/configs/geobenchx.yaml]":valid. The only warning is the expected one: no model server is defined.
tool error. The 21 failures are mistakes in the hand-written references (listed in the
README).
Rollouts
Full real-tools runs (202 tasks × 4 repeats, 808 rollouts each) on three internal
Nemotron 3.5 Super VL checkpoints completed with no rollout failures. Scores and rollouts
are not attached: they are for unreleased checkpoints, and real-tools rollouts contain
data derived from the non-redistributable snapshot.
Compatibility and benchmark impact
None. This is a new resources server and changes no existing code. Simulator scores and
real-tools scores are not comparable; the README says so.
Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).Pre-commit was run on the changed files rather than with
--all-files.