Skip to content

chore(hermes_agent): use pinned upstream Hermes release - #2742

Draft
cwing-nvidia wants to merge 7 commits into
mainfrom
chore/hermes-upstream-release-2672
Draft

cwing-nvidia wants to merge 7 commits into
mainfrom
chore/hermes-upstream-release-2672

Conversation

@cwing-nvidia

@cwing-nvidia cwing-nvidia commented Aug 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Move Gym's Hermes agent from an old fork commit to a reproducible upstream release while preserving the rollout, verification, and observability contracts needed by Gym.

Details

  • Pin upstream Hermes Agent v2026.8.19 / 0.20.5 by release, commit, and installer checksum so the runtime is named, auditable, and reproducible.
  • Install Hermes with its official installer and reconcile the managed environment with the release's frozen uv.lock, avoiding unsupported wheel/sdist installation.
  • Run Hermes in its isolated Python 3.11 environment through a child-process adapter so its dependency set does not conflict with Gym's Python 3.13 server environment.
  • Keep each rollout stateless with an isolated HERMES_HOME, disabled persistence/streaming, bounded concurrency, and child-process signal handling.
  • Use Gym's shared ResponsesConverter for request history and response translation instead of maintaining Hermes-specific trajectory conversion.
  • Project Hermes 0.20.5's deferred tool_call dispatcher into the semantic action it executed, while keeping tool_search, tool_describe, failed pre-dispatch attempts, and the complete raw event stream out of the verifier-facing action list and available to observability.
  • Remove the old Hermes-specific behavior that emitted a synthetic empty assistant message for every tool-only model call; canonical Responses output now uses function_call items for those steps, so turns_used continues to mean assistant-message count while model-call count comes from observability.
  • Preserve Gym observability across the process boundary by collecting dependency-free raw Hermes events in the child and projecting them into canonical AgentObservationBundle records in the parent.
  • Keep training token capture explicitly opt-in and document the runtime, observability, and supported local Hermes terminal backend boundaries.
  • Support the Chat Completions assistant content union in ResponsesConverter, preserving ordered text and refusal parts instead of failing on text lists or silently dropping refusals.

Closes #2672.

Testing

Automated checks

  • pytest responses_api_agents\/hermes_agent\/tests tests\/unit_tests\/test_responses_converter.py -q — 149 passed, including concurrent subprocess isolation, deferred-tool trajectory projection, text-list, list-refusal, and top-level-refusal conversion coverage.
  • pytest responses_api_agents/opencode_sandboxed_agent/tests/test_app.py responses_api_models/litellm_model/tests/test_app.py tests/unit_tests/test_openai_utils.py -q — 149 passed, covering every non-Hermes direct consumer of postprocess_assistant_message_dict plus shared OpenAI schema compatibility.
  • ruff check and ruff format --check on all changed Python files — passed.
  • pre-commit run --all-files — passed all repository hooks without modifying tracked files.

Managed-runtime validation

  • Installed and validated the checksum-pinned Hermes 0.20.5 runtime on macOS, including frozen lock reconciliation.
  • Smoke-tested an empty-cache install in a disposable Linux x86_64 Debian container using Python 3.13. It produced Hermes 0.20.5 at the pinned commit, validated the frozen-lock marker and AIAgent import, and reused the same runtime on a second ensure_hermes() call.
  • Ran three concurrent managed Hermes 0.20.5 child runtimes against a local OpenAI-compatible model stub. All three overlapped, used distinct HERMES_HOME and working directories, returned valid results and observations, exited successfully, removed their temporary directories, left active_processes empty, and left no orphaned runner process. A deterministic regression test now enforces this contract without external credentials.

Real rollouts and comparison

  • Ran a real tool-using Hermes rollout with nvidia/nvidia/nemotron-3.5-lightning; it completed model calls, invoked the local terminal tool, converted the trajectory, verified the response, and returned Gym observation records.
  • Ran the same five math_with_judge examples, model, and concurrency against original main and this branch. Both scored 3/5 and produced the same extracted answer and reward for every example. Original Hermes made 32 model calls and 27 tool calls (three failed); updated Hermes made 10 model calls and five successful tool calls. This is one small behavioral sample, not a performance benchmark.
  • Confirmed turns_used averaged 6.4 on original Hermes and 1.0 on the updated adapter because this repository metric counts assistant message items, not internal model or tool calls. The old custom conversion emitted synthetic empty assistant messages for tool-only steps; shared Responses conversion emits canonical function_call items without empty messages. Updated observability separately recorded all 10 model calls.
  • A stacked Workplace MCP rollout exercised Hermes's deferred-tool mode: the raw runtime used tool_search, tool_describe, and tool_call, while the stored Responses trajectory retained only the successfully executed semantic action. That rollout and its stored-artifact reverification both scored 1.0.

Pending validation

  • Run a real multi-call tool rollout against a self-hosted vLLM server with training token capture enabled and verify complete response reconstruction.

Run Hermes in its official isolated environment so Gym can track a named upstream release without mixing incompatible dependencies.

Signed-off-by: Chris Wing <cwing@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Aug 25, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

Mark the upstream commit and installer checksum as intentional non-secret integrity values so secret scanning can distinguish them from credentials.

Signed-off-by: Chris Wing <cwing@nvidia.com>
Handle text-list and refusal outputs as declared by the Chat Completions schema so Hermes conversion does not lose valid assistant content.

Signed-off-by: Chris Wing <cwing@nvidia.com>
Guard per-rollout Hermes homes and child cleanup under concurrent execution.

Signed-off-by: Chris Wing <cwing@nvidia.com>
@cwing-nvidia
cwing-nvidia marked this pull request as ready for review August 26, 2026 00:53
@cwing-nvidia

Copy link
Copy Markdown
Contributor Author

/claude review

@claude

claude Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Review incomplete: the PR revision changed during review. This review was performed against head 5ca0ae4ea0c74ea8d98992d5d4714eb3c7f3c49d (base 07bb0f53ea2e97d5b0b3c3b63b072f3aae319a69), but the current head is now a43182b9a8155f037501a881c51b460f2aa99f87 (base 129856741a3f67abbd755e4fc76c1e037e7fe244). The final revision check failed, so no inline findings are being posted against a stale diff. Please re-request review to run against the current revision.

Translate upstream deferred-tool dispatcher messages into the executed actions expected by Responses trajectories while leaving raw observability intact.

Signed-off-by: Chris Wing <cwing@nvidia.com>
@cmunley1

Copy link
Copy Markdown
Contributor

could you validate training with tokenid capture for a couple steps?

it seems calls are being marked incomplete maybe due to some harness generated calls that trigger mark_incomplete

@cmunley1

Copy link
Copy Markdown
Contributor

/claude review

@cmunley1

Copy link
Copy Markdown
Contributor

could you validate training with tokenid capture for a couple steps?

it seems calls are being marked incomplete maybe due to some harness generated calls that trigger mark_incomplete

This fix might need to go into tokenid capture @ananthsub

I am not certain it is the cause but it seems so:

The main chain is retained correctly, but then a separate “capture incomplete” flag masks the whole rollout afterward.
Exact sequence:

  1. Hermes sends 20 model-discovery requests through the token-capture URL, including /v1/models, /api/tags, /v1/props, /version, and /api/show.
  2. Gym recognizes only generation endpoints such as /v1/chat/completions.
  3. Each discovery request is therefore recorded as an unidentified incomplete capture.
  4. Hermes then makes one valid generation call. Gym captures its token IDs and reconstructs the main chain successfully:
    • chains=1
    • roots=1
    • quarantined_calls=0
    • delivered_fraction=1.0
  5. Despite that valid chain, consumer.py sees the earlier global snapshot.incomplete flag and unconditionally sets mask_sample=True.

@claude

claude Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

SHIP WITH CARE

Reviewed the full diff. This refactors the Hermes agent from an in-process AIAgent import to a managed-subprocess boundary (isolated Python 3.11 runtime, JSON request/response, per-rollout HERMES_HOME), and generalizes postprocess_assistant_message_dict to handle list-form content + refusal parts. The subprocess isolation, SIGTERM dispatch over active_processes, cancellation cleanup (SIGTERM → wait → kill), and observability two-layer bridge are well-constructed and well-tested. errors="replace" on subprocess stderr and the checksum-pinned installer are correct per repo conventions.

One conditional correctness finding (inline, app.py:109): project_hermes_response_messages() runs over the full message list before the messages[n_input:] slice, so any projection-deleted message in the echoed input prefix shifts the slice and drops real output items. The common single-turn eval path (n_input=1) is safe; the exposure is resumed/multi-turn rollouts whose history carries projectable artifacts. Worth confirming before merge that no realistic input prefix hits projection.

Other notes (author call, not blocking):

  • HermesAgentConfig.token_id_capture redefines the field already declared on BaseResponsesAPIAgentConfig with the same default — harmless, and the explanatory comment is useful, but it is redundant.
  • _run_hermes_subprocess reads the whole trajectory into memory and does no server-side concurrency bound itself; concurrency is gated upstream in run() via self.sem, and _create_response is also reachable directly from responses() without the semaphore. Unbounded concurrent subprocesses on the direct /v1/responses path is a latent operability concern if that path is driven at scale, but eval/training goes through run(), so low priority.

Verifier/scoring logic (run() turn counting, finished_naturally) is unchanged in substance and reads correctly. Async paths use asyncio.create_subprocess_exec and awaitable process I/O throughout — no httpx, no blocking ray.get. Test coverage for the new surfaces (subprocess isolation, request_overrides, trajectory projection, refusal conversion) is real and asserts behavior.

Preserve every new output item when resumed input history contains Hermes-internal dispatch messages that projection removes.

Signed-off-by: Chris Wing <cwing@nvidia.com>
@cwing-nvidia cwing-nvidia changed the title chore(hermes_agent): use pinned upstream CLI release chore(hermes_agent): use pinned upstream Hermes release Aug 26, 2026
@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Aug 26, 2026
@cwing-nvidia

Copy link
Copy Markdown
Contributor Author

/claude review

@claude

claude Bot commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

SHIP WITH CARE — no blockers found. Two changes, both well-tested; the risk is concentrated in blast radius, not defects.

1. responses_converter.postprocess_assistant_message_dict (shared, all model servers)
The rewrite correctly flattens list-form assistant content, preserves interleaved refusal parts, and handles top-level refusal. Verified the token-capture tail (response_output[-1] → training variant) still can't IndexError: has_empty_output guarantees at least one message item whenever there are no tool calls. Callers (litellm_model, azure_openai_model, opencode_sandboxed_agent) only pass role == "assistant", so role=message_dict.get("role") stays valid.

  • NOTE (defense-in-depth): the loop now hard-raises ValueError/TypeError on unrecognized content-part types instead of coercing. For the common string-content path this is unchanged (no raise). But a provider that emits a novel assistant content part (e.g. an inline reasoning/thinking part) via litellm would now surface as a 500 from postprocess_chat_response. Prior code would also have failed (pydantic error on text=<list>), so this is fail-loud, not a regression — flagging only because it's on the shared model-server output path.

2. hermes_agent → managed subprocess
Async handling is correct: asyncio.create_subprocess_exec + await process.communicate(), CancelledError escalates SIGTERM→kill with a bounded wait, stderr decoded errors="replace", per-rollout TemporaryDirectory + isolated HERMES_HOME (test TestManagedSubprocessIsolation confirms distinct homes and cleanup). The SIGTERM dispatcher over active_processes is the right analogue to the old per-agent set. Installer download is SHA256-pinned with an flock guard and idempotent _is_ready() check.

Worth an explicit call-out for the training team, since it's a deliberate contract change, not a bug:

  • Inline token ids are gone from the Hermes agent response. The old path built NeMoGymResponseOutputMessageForTraining with prompt_token_ids/generation_token_ids/routed_experts pulled from Hermes message dicts; the new path uses ResponsesConverter(return_token_id_information=False), so training tokens now flow only through the out-of-band token_id_capture sink + rebuild_response. This is documented in the README and is inherent to upstream 0.20.5 (which doesn't emit tokens inline over the JSON boundary), but any training config that relied on inline token fields from this agent must now set token_id_capture.enabled + the agent-level token_id_capture: true. Confirm the RLHF collectors are on the reconstruction path before running Hermes training.

  • NOTE: runner.py pokes upstream private attrs (_persist_disabled, _disable_streaming) on AIAgent. Fragile across Hermes versions, but pinned to commit fcbd107… and guarded by test_upstream_release_api_matches_adapter. If the pin bumps, re-verify those attrs exist.

  • NOTE (operability): subprocess-per-rollout is materially heavier than the old thread-per-rollout (full 3.11 interpreter + Hermes deps each), still bounded only by concurrency (default 32) via the outer run() semaphore. On memory-constrained nodes, 32 concurrent Hermes interpreters may pressure RAM where 32 threads did not — size concurrency accordingly.

Tests are substantive (assert real projection/isolation/conversion behavior, not pass-throughs). No async-client or ray.get violations introduced.

@ananthsub

ananthsub commented Aug 26, 2026 •

Copy link
Copy Markdown
Contributor

could you validate training with tokenid capture for a couple steps?

it seems calls are being marked incomplete maybe due to some harness generated calls that trigger mark_incomplete

@cmunley1 it is, also filed at #2678 - auxiliary calls to endpoints not supported by the gym model server are treated as failures that mask the whole rollout. i will send out a fix

@anwithk
anwithk requested a review from ananthsub August 26, 2026 16:45
@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer labels Oct 20, 2026
@yaoyu-33
yaoyu-33 marked this pull request as draft November 18, 2026 02:16
@yaoyu-33

Copy link
Copy Markdown
Contributor

We’re doing some cleanup of the open PR list. This PR has had no activity for over a month, so I’ve moved it back to draft.

If this change is still needed, please resolve any merge conflicts and mark the PR ready for review again when it’s ready to merge. Thanks!

@github-actions github-actions Bot removed the sla:review-overdue Review response is over the one-business-day SLA label Nov 18, 2026

This branch was previously deployed

1 inactive deployment
public — 608ea713 Deployed Aug 26, 2026 by copy-pr-bot[bot] via release / finalize / notify #1859
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:agent Agent harnesses and Responses API agent behavior feature New capabilities, enhancements, or enablement work needs-review PR is ready for code review and waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(hermes_agent): evaluate replacing the fork commit with a named upstream release

5 participants