Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,8 @@ env.yaml
# Claude Code agent runtime artifacts
**/.claude_node/

# cvdp generic agent provisioned harness deps (portable python/node + CLIs, built at runtime)
# generic agents' provisioned harness deps (portable python/node + CLIs, built at runtime)
responses_api_agents/cvdp_agent/deps/
responses_api_agents/gdpval_agent/deps/
resources_servers/cvdp/data/gym_agentic_code_generation_no_commercial.jsonl
resources_servers/cvdp/data/gym_cvdp_nonagentic_code_generation_no_commercial.jsonl
6 changes: 4 additions & 2 deletions nemo_gym/openai_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -1147,8 +1147,10 @@ class NeMoGymChatCompletionAssistantMessageForTrainingParam(


class NeMoGymChatCompletionToolMessageParam(ChatCompletionToolMessageParam):
# Override the iterable which is annoying to work with.
content: Required[Union[str, List[NeMoGymChatCompletionContentPartTextParam]]]
# Override the iterable which is annoying to work with. Text-only content rejects the whole
# conversation the first time a vision tool returns a picture, so accept what a user message
# accepts.
content: Required[Union[str, List[NeMoGymChatCompletionContentPartParam]]]


class NeMoGymFunctionToolParam(FunctionToolParam):
Expand Down
62 changes: 62 additions & 0 deletions responses_api_agents/gdpval_agent/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# GDPVal agent

Runs the [GDPVal](https://huggingface.co/datasets/openai/gdpval) benchmark with a pluggable
harness executing inside the GDPVal container, and scores the files the harness produced.

The benchmark's own agent, `stirrup_agent`, drives the model itself. This agent instead
provisions the container, stages the task's reference files, runs another Gym agent inside
it, and collects the deliverables. That makes GDPVal usable as a harness comparison.

## How it differs from stirrup_agent

`stirrup_agent` runs its loop on the host and sends only its shell tool into the container.
Here the whole harness process runs inside the container, so the harness needs no sandbox of
its own and its file tools write directly into the workspace.

The harness has no tool for declaring which files are its deliverables. Instead the prompt
names an output directory and only that directory is collected. If it is empty, files under
the workspace that are newer than the start marker are collected instead.

## Setup

Build the container:

```bash
apptainer build gdpval.sif responses_api_agents/stirrup_agent/containers/gdpval.def
```

Environment:

| Variable | Purpose |
| --- | --- |
| `GDPVAL_CONTAINER_PATH` | Absolute path to the `.sif`. |
| `PERSIST_DELIVERABLES_DIR` | Absolute path where deliverables are written for scoring. |
| `HF_TOKEN` | Downloading the dataset and its reference files. |
| `JUDGE_API_KEY` | Judge endpoint key. Use an `sk-` key; `nvapi-` keys reject multimodal payloads. |
| `JUDGE_BASE_URL`, `JUDGE_GPT_MODEL`, `JUDGE_GEMINI_MODEL`, `JUDGE_CLAUDE_MODEL` | Judge overrides. |

## Run

```bash
gym eval prepare --benchmark gdpval/hermes
gym eval run \
--benchmark gdpval/hermes \
--model-type vllm_model \
--split benchmark \
--output results/gdpval_hermes.jsonl
```

## Swapping the harness

Change `agent_server_module`, `agent_server_class` and `agent_config_class` in
`configs/gdpval_agent.yaml`. The dependency prefix is built from
`responses_api_agents/<harness>/scripts/<harness>_deps.sh`, so a harness needs that script to
exist. Set `agent_kwargs` to whatever that harness's config class accepts.

## Checking a run

The deliverables that reached the judge are on disk at
`$PERSIST_DELIVERABLES_DIR/task_<task_id>/repeat_<n>/`. They should be real documents, which
`file` will confirm, and they should be flat: the scorer does not descend into
subdirectories. A reward with no files there means the judge scored the model's closing
message rather than its work.
Empty file.
Loading