Skip to content

fix(telemetry): give each server its own service.instance.id - #5100

Merged
rrs45 merged 2 commits into
mainfrom
raj/gym-per-server-instance-id
Nov 5, 2026
Merged

rrs45 merged 2 commits into
mainfrom
raj/gym-per-server-instance-id

Conversation

@rrs45

@rrs45 rrs45 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Gives each Gym server process its own service.instance.id, so per-server traces are usable on large fleets.

Closes #5048

fix(telemetry): give each server its own service.instance.id

Gym never set service.instance.id, so nemo-lens derived one as f"{config.run_id}-rank{rank}". Since run_id is shared fleet-wide by design and every Gym server correctly reports rank 0 of a world of 1, every server in a job resolved to the same "<run id>-rank0". On a NeMo-RL run that is ~100 processes sharing a single instance id: the fleet is visible but no individual server is, so you cannot tell which server handled a given rollout or which one is misbehaving.

This names the process explicitly instead of leaving the derivation to lens. The server name separates servers and the PID separates one server's Uvicorn workers, behind a random suffix that is what actually satisfies the semantic convention — neither readable part is unique on its own, since a PID repeats across hosts and across container PID namespaces and the run id is shared. Supplying the attribute also suppresses the lens derivation, which only fires when no identity was given. nv.dl.rank is deliberately unchanged, along with the test asserting it.

Done here rather than by handing out per-server ranks from the supervisor because servers are spawned from several places (the CLI, dev, the Ray gateway, the process supervisor) and can also be run standalone; a rank protocol would have to reach all of them.

test(orchestration): pin the Ray Serve gateway's freedom from nemo_gym imports

The Slurm launcher base64-embeds ray_serve_gateway.py into the vLLM container and runs it with only aiohttp and Ray installed, so this path needs no driver.gym_install. Any nemo_gym import in that file raises ModuleNotFoundError before argv is parsed and the model service never becomes available. Nothing imports nemo_gym there today; this guards that, since the file reads like ordinary Gym source and the constraint is invisible from inside it.

What changed since the last review

The fastapi_operation_spans unit has been dropped, per @ananthsub's review. It had no effect on Gym's servers in any version Gym allows: fastapi!=0.142.* excludes the one release that double-exports, the lockfile pins 0.135.2 where FastAPI.__init__ has no telemetry parameter at all, and from 0.143 FastAPI stands down when it sees the OTel instrumentor. The measured span volume came from NeMo-RL's pinned Gym, whose server venvs lack opentelemetry-instrumentation-fastapi; NVIDIA-NeMo/RL#4894 is where that belongs. The duplicated gateway helper went with it, leaving only the import guard above.

The instance-id docs were also corrected on two points the review raised: a service.instance.id in OTEL_RESOURCE_ATTRIBUTES is replaced, not kept (lens ranks caller attributes above environment ones, and Gym now always passes one), and the run id prefixes the identity only when the process was configured with one, since lens mints nemo.run.id later inside setup_telemetry.

Validation

Run against the final diff, rebased onto main at 80cc55d:

python -m pytest tests/unit_tests/telemetry/ tests/unit_tests/test_slurm_script.py tests/unit_tests/test_ray_serve_gateway.py
# 742 passed, 1 skipped, 3 failed
pre-commit run --from-ref origin/main --to-ref HEAD   # all hooks passed

The 3 failures (test_connection_pool.py::test_multi_destination_waits_are_attributed_to_the_binding_limit and both tests in test_memory.py) reproduce identically on a clean origin/main worktree and are unrelated to this change. They appear to be macOS-only, which is why the review did not see them; test_call_sites.py::test_supervisor_is_the_only_writer_of_the_active_servers_gauge, listed as failing previously, now passes on main.

Rollouts

N/A — no environment or agent behavior changes. The change affects only a resource attribute; request handling, rollout execution, and scoring paths are untouched.

Compatibility and benchmark impact

No migration required and no benchmark-result effects.

service.instance.id is newly populated where it was previously derived by lens. Dashboards or queries that relied on the old shared "<run id>-rank0" value to group an entire fleet will now see one series per server process; grouping by nemo.run.id gives the previous behavior.

A deployment that sets service.instance.id in OTEL_RESOURCE_ATTRIBUTES no longer keeps that value, because Gym's servers are spawned into one shared environment and a single inherited id would name all of them identically. Passing a resource attribute programmatically still takes precedence.

Checklist

  • I have read the contributing guidelines.
  • The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
  • Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
  • Documentation added or updated, or N/A with justification. (The observability configuration page documents the new attribute and the environment-variable precedence.)
  • Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
  • New source files include the required Apache-2.0 SPDX header. (N/A — no new source files; all touched files already existed.)
  • All commits have DCO sign-off (git commit -s) (so the DCO check passes).

@copy-pr-bot

copy-pr-bot Bot commented Oct 26, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci svcnvidia-nemo-ci added area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working labels Oct 26, 2026
@rrs45
rrs45 marked this pull request as ready for review October 26, 2026 22:45
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the needs-review PR is ready for code review and waiting on a reviewer label Oct 26, 2026
@rrs45
rrs45 force-pushed the raj/gym-per-server-instance-id branch from 76a3cec to 5c99634 Compare October 26, 2026 22:50
@rrs45

rrs45 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 5c99634

@iguehring-nvidia

Copy link
Copy Markdown
Contributor

/review

@svcnvidia-nemo-ci svcnvidia-nemo-ci left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HOLD

Found one BLOCKER affecting standalone Ray Serve deployments and one RISK in fleet-wide process identity. Both include narrow corrections and preservation checks inline.

NOTE (suggestion) — Add the public telemetry changes to the user guide. The new setting is documented only in the developer design note. The published configuration guide still omits the flag, its environment variable, and the changed instance identity. Users following that guide cannot discover the opt-in or the new per-process grouping behavior.

Could you update its field, environment-variable, and resource-attribute sections? Include fastapi_operation_spans with default false, NEMO_GYM_OTEL_FASTAPI_OPERATION_SPANS, and this minimal opt-in example:

telemetry:
  enabled: true
  fastapi_operation_spans: true

Explain the generated per-process service.instance.id, while retaining shared nemo.run.id for fleet grouping. Preserve the existing environment-over-YAML precedence and distinguish operation spans from HTTP request spans. Acceptance (proposed, not executed): verify those three documentation sections contain the new contracts and that the example matches the config schema.

All ten changed files and their directly affected configuration, startup, launcher, and test paths were inspected. Configuration/helper coverage was independently completed after its delegated report was unavailable. This was a static review; no tests were executed, and the external FastAPI/nemo-lens implementations were not mounted for independent runtime verification.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

from fastapi import FastAPI, Request, Response
from ray import serve

from nemo_gym.telemetry.setup import fastapi_telemetry_kwargs

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BLOCKER — Preserve the gateway's standalone deployment contract

The supported Slurm launcher copies only this script into the vLLM container and installs aiohttp/Ray as needed; it deliberately does not install Gym. In a serving image without nemo_gym, this new unconditional import raises ModuleNotFoundError before argument parsing or server startup—even with telemetry disabled.

This breaks explicit use_ray_serve: true and the automatic cross-node selection, for example TP8 × PP2 with two instances on four eight-GPU nodes. The selected model service never becomes available. The base script had no Gym imports.

Could you keep this shipped script self-contained? Replacing this import with a local helper preserves the current shared helper contract without installing the entire Gym stack:

Suggested change
from nemo_gym.telemetry.setup import fastapi_telemetry_kwargs
def fastapi_telemetry_kwargs() -> dict:
import inspect
if "telemetry" not in inspect.signature(FastAPI.__init__).parameters:
return {}
enabled = os.environ.get("NEMO_GYM_OTEL_FASTAPI_OPERATION_SPANS", "").strip().lower() in (
"1", "true", "yes", "on"
)
return {"telemetry": {"operation_spans": enabled}}

This uses existing imports plus the standard library. Verify the suggestion before adoption; keep the other five Gym-owned constructor integrations unchanged.

Preservation / acceptance (proposed, not executed): copy the gateway outside the checkout and run python3 ray_serve_gateway.py --help with its serving dependencies installed but Gym unavailable; expect exit 0, not an import failure. For constructors supporting telemetry, unset/blank/false env must produce operation spans false and 1/true/yes/on must produce true; older constructors must receive no telemetry keyword. Retain the embedded-source/no-clone assertions, the single-node quoting assertion that org/my model remains one argument, and the multi-node chain assertions that the complete launch chain remains one opaque command.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d35a1d1. Dropped the nemo_gym.telemetry.setup import and resolved the setting locally from inspect plus the imports the gateway already has, so the embedded file stays runnable where only aiohttp and Ray are installed. The five Gym-owned constructors still use the shared helper.

Two tests pin the contract, both in test_slurm_script.py so they run without Ray: test_embedded_gateway_imports_nothing_from_nemo_gym parses the source and fails on any nemo_gym import, including one deferred into a function, and test_embedded_gateway_reads_the_operation_spans_variable_gym_exports reads the duplicated env var name back out of the source and compares it to _ENV_FIELD_MAP, so the copy cannot drift. The truthy parsing itself is covered by a parametrized test in test_ray_serve_gateway.py.

Comment thread nemo_gym/telemetry/setup.py Outdated
workers. Both are omitted when absent rather than left as an empty segment, so a
standalone server with no run id still gets ``<name>-<pid>`` instead of ``--<pid>``.
"""
return "-".join(part for part in (run_id, server_name, str(os.getpid())) if part)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RISK — Make instance identity unique beyond the local PID namespace

The new scheme still collides when processes on different hosts or in separate container PID namespaces have the same local PID. Minimal example: two independent servers with run job-1, config name weather, and local PID 42 both pass service.instance.id=job-1-weather-42 to nemo-lens. The run ID is deliberately shared fleet-wide, and server initialization supplies the config name, not a host-unique identity. Thus this partially fixes the old shared identity but does not establish the promised per-process distinction across a distributed fleet.

Could you retain the readable prefix and append a process-lifetime nonce? uuid4 is already imported, and the initialization guard ensures the helper is called only once per Gym-owned initialization:

Suggested change
return "-".join(part for part in (run_id, server_name, str(os.getpid())) if part)
return "-".join(part for part in (run_id, server_name, str(os.getpid()), uuid4().hex) if part)

Verify this suggestion before adoption. Update this helper's format description and the new exact-format test accordingly. Keep the subsequent caller-attribute merge so explicit identities still win; do not change rank/world size, service-name suffixing/opt-out, or externally owned provider reuse.

Preservation / acceptance (proposed, not executed): capture attributes from two independent initialization lifecycles with identical run/name and mocked PID 42; generated IDs must differ. Within one lifecycle, repeated initialization must retain the same handle and identity. Different server names and different worker PIDs must remain distinguishable. Check the readable prefix followed by a valid nonce rather than one exact full string, and check absent run/name yields a nonempty identity without empty prefix segments. Retain the existing assertions for caller identity (chosen-by-the-caller), rank/world size (0/1), and service-name suffixing/opt-out.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 55bb42b. _service_instance_id now appends a random suffix, so the identity no longer rests on the PID being unique: <run id>-<server name>-<pid>-<nonce>.

One deviation from the suggestion, flag it if you disagree: eight hex digits of a uuid4 rather than the full 32, matching the truncated run id a few lines up. This only has to separate the processes of one run, not be unique across every run ever submitted, and it keeps the identity readable. init_telemetry sets _INITIALISED under _INIT_LOCK and returns early afterwards, so the suffix is drawn once and is stable for the process lifetime.

Unchanged as asked: the readable prefix, the merge that lets a caller's explicit service.instance.id win, rank/world size, service-name suffixing, and externally owned provider reuse.

The format test now pins the prefix and requires the suffix to be hex instead of the whole string. Added test_two_processes_sharing_a_run_a_name_and_a_pid_still_differ for the collision itself, and test_an_instance_id_stays_nonempty_without_a_run_or_a_name for the no-leading-separator case.

@github-actions github-actions Bot added the sla:review-overdue Review response is over the one-business-day SLA label Oct 28, 2026
@rrs45
rrs45 force-pushed the raj/gym-per-server-instance-id branch from 5c99634 to 55bb42b Compare November 3, 2026 16:01
@rrs45

rrs45 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 55bb42b

rrs45 added 2 commits November 5, 2026 07:23
Every Gym server reports rank 0 of a world of 1, which is correct -- each is
its own single-process world, and reporting a rank is what suppresses
nemo-lens's no-rank warning. But Gym never set service.instance.id, and lens
fills that gap by deriving one:

    if rank is not None and config.run_id and not explicit_identity:
        attrs["service.instance.id"] = f"{config.run_id}-rank{rank}"

run_id is shared fleet-wide by design, so every server in the job resolved to
the same "<run id>-rank0". On a NeMo-RL run that is ~100 processes sharing one
instance id -- the fleet is visible but no single server is, and you cannot
tell which one served a given rollout or which one is misbehaving.

Name the process instead of leaving the derivation to lens: the server name
separates the servers and the PID separates one server's Uvicorn workers,
behind a random suffix that is what actually satisfies the semantic
convention. Neither readable part is unique on its own -- a PID repeats across
hosts and across container PID namespaces, and the run id is shared.

Done here rather than by handing out per-server ranks from the supervisor
because servers are spawned from several places (the CLI, dev, the Ray
gateway, the process supervisor) and can be run standalone; a rank protocol
would have to reach all of them. nv.dl.rank is deliberately unchanged, along
with the test asserting it.

Because lens ranks caller attributes above environment ones, this replaces a
service.instance.id inherited from OTEL_RESOURCE_ATTRIBUTES. That is what a
fleet needs, since Gym's servers share one spawned environment and a single
inherited value would name them all identically; code embedding Gym can still
pass its own id through resource_attributes, which still wins. The docs page
and the tests say so explicitly.

The run id prefixes the identity only when the process was configured with
one. A standalone server gets <name>-<pid>-<nonce>: lens mints its nemo.run.id
later, inside setup_telemetry, which is after this id is built.

Signed-off-by: Raj Singh <rajsin@nvidia.com>
…m imports

The Slurm launcher base64-embeds ray_serve_gateway.py into the vLLM container
and runs it with only aiohttp and Ray installed -- deliberately, so this path
needs no driver.gym_install. Any nemo_gym import in that file therefore raises
ModuleNotFoundError before argv is parsed, and the selected model service
never becomes available, breaking explicit `use_ray_serve: true` and the
automatic cross-node selection.

Nothing imports nemo_gym there today. This guards that, since the file reads
like ordinary Gym source and the constraint is invisible from inside it.

Parsed rather than grepped so an import deferred into a function counts too:
deferring one only moves the failure to the first call. Parsed rather than
imported because importing the gateway needs Ray, an optional extra, and a
guard that only runs where one extra happens to be installed is worth nothing.
It lives in test_slurm_script.py for the same reason, beside the test that
pins the embedded source itself.

Signed-off-by: Raj Singh <rajsin@nvidia.com>
@rrs45

rrs45 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 475fecc

@rrs45
rrs45 added this pull request to the merge queue Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42125685293

Merged via the queue into main with commit 3a19fea Nov 5, 2026
121 of 124 checks passed
@rrs45
rrs45 deleted the raj/gym-per-server-instance-id branch November 5, 2026 23:12

This branch was previously deployed

1 inactive deployment
public — 475fecc6 Deployed Nov 5, 2026 by copy-pr-bot[bot] via release / finalize / notify #10334
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core Shared APIs, servers, telemetry, health, and registries bug Something isn't working needs-review PR is ready for code review and waiting on a reviewer sla:review-overdue Review response is over the one-business-day SLA

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Telemetry: every server reports rank 0, so they share one service.instance.id

4 participants