Repository navigation
fix(telemetry): give each server its own service.instance.id - #5100
Conversation
76a3cec to
5c99634
Compare
|
/ok to test 5c99634 |
|
/review |
svcnvidia-nemo-ci
left a comment
There was a problem hiding this comment.
HOLD
Found one BLOCKER affecting standalone Ray Serve deployments and one RISK in fleet-wide process identity. Both include narrow corrections and preservation checks inline.
NOTE (suggestion) — Add the public telemetry changes to the user guide. The new setting is documented only in the developer design note. The published configuration guide still omits the flag, its environment variable, and the changed instance identity. Users following that guide cannot discover the opt-in or the new per-process grouping behavior.
Could you update its field, environment-variable, and resource-attribute sections? Include fastapi_operation_spans with default false, NEMO_GYM_OTEL_FASTAPI_OPERATION_SPANS, and this minimal opt-in example:
telemetry:
enabled: true
fastapi_operation_spans: trueExplain the generated per-process service.instance.id, while retaining shared nemo.run.id for fleet grouping. Preserve the existing environment-over-YAML precedence and distinguish operation spans from HTTP request spans. Acceptance (proposed, not executed): verify those three documentation sections contain the new contracts and that the example matches the config schema.
All ten changed files and their directly affected configuration, startup, launcher, and test paths were inspected. Configuration/helper coverage was independently completed after its delegated report was unavailable. This was a static review; no tests were executed, and the external FastAPI/nemo-lens implementations were not mounted for independent runtime verification.
Note: formal approval is not available for this request, so this review is posted as a comment only.
Review generated by codex. This was a full review. Request a full review with /review reset=true.
| from fastapi import FastAPI, Request, Response | ||
| from ray import serve | ||
|
|
||
| from nemo_gym.telemetry.setup import fastapi_telemetry_kwargs |
There was a problem hiding this comment.
BLOCKER — Preserve the gateway's standalone deployment contract
The supported Slurm launcher copies only this script into the vLLM container and installs aiohttp/Ray as needed; it deliberately does not install Gym. In a serving image without nemo_gym, this new unconditional import raises ModuleNotFoundError before argument parsing or server startup—even with telemetry disabled.
This breaks explicit use_ray_serve: true and the automatic cross-node selection, for example TP8 × PP2 with two instances on four eight-GPU nodes. The selected model service never becomes available. The base script had no Gym imports.
Could you keep this shipped script self-contained? Replacing this import with a local helper preserves the current shared helper contract without installing the entire Gym stack:
| from nemo_gym.telemetry.setup import fastapi_telemetry_kwargs | |
| def fastapi_telemetry_kwargs() -> dict: | |
| import inspect | |
| if "telemetry" not in inspect.signature(FastAPI.__init__).parameters: | |
| return {} | |
| enabled = os.environ.get("NEMO_GYM_OTEL_FASTAPI_OPERATION_SPANS", "").strip().lower() in ( | |
| "1", "true", "yes", "on" | |
| ) | |
| return {"telemetry": {"operation_spans": enabled}} |
This uses existing imports plus the standard library. Verify the suggestion before adoption; keep the other five Gym-owned constructor integrations unchanged.
Preservation / acceptance (proposed, not executed): copy the gateway outside the checkout and run python3 ray_serve_gateway.py --help with its serving dependencies installed but Gym unavailable; expect exit 0, not an import failure. For constructors supporting telemetry, unset/blank/false env must produce operation spans false and 1/true/yes/on must produce true; older constructors must receive no telemetry keyword. Retain the embedded-source/no-clone assertions, the single-node quoting assertion that org/my model remains one argument, and the multi-node chain assertions that the complete launch chain remains one opaque command.
There was a problem hiding this comment.
Done in d35a1d1. Dropped the nemo_gym.telemetry.setup import and resolved the setting locally from inspect plus the imports the gateway already has, so the embedded file stays runnable where only aiohttp and Ray are installed. The five Gym-owned constructors still use the shared helper.
Two tests pin the contract, both in test_slurm_script.py so they run without Ray: test_embedded_gateway_imports_nothing_from_nemo_gym parses the source and fails on any nemo_gym import, including one deferred into a function, and test_embedded_gateway_reads_the_operation_spans_variable_gym_exports reads the duplicated env var name back out of the source and compares it to _ENV_FIELD_MAP, so the copy cannot drift. The truthy parsing itself is covered by a parametrized test in test_ray_serve_gateway.py.
| workers. Both are omitted when absent rather than left as an empty segment, so a | ||
| standalone server with no run id still gets ``<name>-<pid>`` instead of ``--<pid>``. | ||
| """ | ||
| return "-".join(part for part in (run_id, server_name, str(os.getpid())) if part) |
There was a problem hiding this comment.
RISK — Make instance identity unique beyond the local PID namespace
The new scheme still collides when processes on different hosts or in separate container PID namespaces have the same local PID. Minimal example: two independent servers with run job-1, config name weather, and local PID 42 both pass service.instance.id=job-1-weather-42 to nemo-lens. The run ID is deliberately shared fleet-wide, and server initialization supplies the config name, not a host-unique identity. Thus this partially fixes the old shared identity but does not establish the promised per-process distinction across a distributed fleet.
Could you retain the readable prefix and append a process-lifetime nonce? uuid4 is already imported, and the initialization guard ensures the helper is called only once per Gym-owned initialization:
| return "-".join(part for part in (run_id, server_name, str(os.getpid())) if part) | |
| return "-".join(part for part in (run_id, server_name, str(os.getpid()), uuid4().hex) if part) |
Verify this suggestion before adoption. Update this helper's format description and the new exact-format test accordingly. Keep the subsequent caller-attribute merge so explicit identities still win; do not change rank/world size, service-name suffixing/opt-out, or externally owned provider reuse.
Preservation / acceptance (proposed, not executed): capture attributes from two independent initialization lifecycles with identical run/name and mocked PID 42; generated IDs must differ. Within one lifecycle, repeated initialization must retain the same handle and identity. Different server names and different worker PIDs must remain distinguishable. Check the readable prefix followed by a valid nonce rather than one exact full string, and check absent run/name yields a nonempty identity without empty prefix segments. Retain the existing assertions for caller identity (chosen-by-the-caller), rank/world size (0/1), and service-name suffixing/opt-out.
There was a problem hiding this comment.
Done in 55bb42b. _service_instance_id now appends a random suffix, so the identity no longer rests on the PID being unique: <run id>-<server name>-<pid>-<nonce>.
One deviation from the suggestion, flag it if you disagree: eight hex digits of a uuid4 rather than the full 32, matching the truncated run id a few lines up. This only has to separate the processes of one run, not be unique across every run ever submitted, and it keeps the identity readable. init_telemetry sets _INITIALISED under _INIT_LOCK and returns early afterwards, so the suffix is drawn once and is stable for the process lifetime.
Unchanged as asked: the readable prefix, the merge that lets a caller's explicit service.instance.id win, rank/world size, service-name suffixing, and externally owned provider reuse.
The format test now pins the prefix and requires the suffix to be hex instead of the whole string. Added test_two_processes_sharing_a_run_a_name_and_a_pid_still_differ for the collision itself, and test_an_instance_id_stays_nonempty_without_a_run_or_a_name for the no-leading-separator case.
5c99634 to
55bb42b
Compare
|
/ok to test 55bb42b |
Every Gym server reports rank 0 of a world of 1, which is correct -- each is
its own single-process world, and reporting a rank is what suppresses
nemo-lens's no-rank warning. But Gym never set service.instance.id, and lens
fills that gap by deriving one:
if rank is not None and config.run_id and not explicit_identity:
attrs["service.instance.id"] = f"{config.run_id}-rank{rank}"
run_id is shared fleet-wide by design, so every server in the job resolved to
the same "<run id>-rank0". On a NeMo-RL run that is ~100 processes sharing one
instance id -- the fleet is visible but no single server is, and you cannot
tell which one served a given rollout or which one is misbehaving.
Name the process instead of leaving the derivation to lens: the server name
separates the servers and the PID separates one server's Uvicorn workers,
behind a random suffix that is what actually satisfies the semantic
convention. Neither readable part is unique on its own -- a PID repeats across
hosts and across container PID namespaces, and the run id is shared.
Done here rather than by handing out per-server ranks from the supervisor
because servers are spawned from several places (the CLI, dev, the Ray
gateway, the process supervisor) and can be run standalone; a rank protocol
would have to reach all of them. nv.dl.rank is deliberately unchanged, along
with the test asserting it.
Because lens ranks caller attributes above environment ones, this replaces a
service.instance.id inherited from OTEL_RESOURCE_ATTRIBUTES. That is what a
fleet needs, since Gym's servers share one spawned environment and a single
inherited value would name them all identically; code embedding Gym can still
pass its own id through resource_attributes, which still wins. The docs page
and the tests say so explicitly.
The run id prefixes the identity only when the process was configured with
one. A standalone server gets <name>-<pid>-<nonce>: lens mints its nemo.run.id
later, inside setup_telemetry, which is after this id is built.
Signed-off-by: Raj Singh <rajsin@nvidia.com>
…m imports The Slurm launcher base64-embeds ray_serve_gateway.py into the vLLM container and runs it with only aiohttp and Ray installed -- deliberately, so this path needs no driver.gym_install. Any nemo_gym import in that file therefore raises ModuleNotFoundError before argv is parsed, and the selected model service never becomes available, breaking explicit `use_ray_serve: true` and the automatic cross-node selection. Nothing imports nemo_gym there today. This guards that, since the file reads like ordinary Gym source and the constraint is invisible from inside it. Parsed rather than grepped so an import deferred into a function counts too: deferring one only moves the failure to the first call. Parsed rather than imported because importing the gateway needs Ray, an optional extra, and a guard that only runs where one extra happens to be installed is worth nothing. It lives in test_slurm_script.py for the same reason, beside the test that pins the embedded source itself. Signed-off-by: Raj Singh <rajsin@nvidia.com>
1146733 to
475fecc
Compare
|
/ok to test 475fecc |
|
🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/42125685293 |
What does this PR do?
Gives each Gym server process its own
service.instance.id, so per-server traces are usable on large fleets.Closes #5048
fix(telemetry): give each server its own service.instance.idGym never set
service.instance.id, so nemo-lens derived one asf"{config.run_id}-rank{rank}". Sincerun_idis shared fleet-wide by design and every Gym server correctly reports rank 0 of a world of 1, every server in a job resolved to the same"<run id>-rank0". On a NeMo-RL run that is ~100 processes sharing a single instance id: the fleet is visible but no individual server is, so you cannot tell which server handled a given rollout or which one is misbehaving.This names the process explicitly instead of leaving the derivation to lens. The server name separates servers and the PID separates one server's Uvicorn workers, behind a random suffix that is what actually satisfies the semantic convention — neither readable part is unique on its own, since a PID repeats across hosts and across container PID namespaces and the run id is shared. Supplying the attribute also suppresses the lens derivation, which only fires when no identity was given.
nv.dl.rankis deliberately unchanged, along with the test asserting it.Done here rather than by handing out per-server ranks from the supervisor because servers are spawned from several places (the CLI, dev, the Ray gateway, the process supervisor) and can also be run standalone; a rank protocol would have to reach all of them.
test(orchestration): pin the Ray Serve gateway's freedom from nemo_gym importsThe Slurm launcher base64-embeds
ray_serve_gateway.pyinto the vLLM container and runs it with only aiohttp and Ray installed, so this path needs nodriver.gym_install. Anynemo_gymimport in that file raisesModuleNotFoundErrorbefore argv is parsed and the model service never becomes available. Nothing importsnemo_gymthere today; this guards that, since the file reads like ordinary Gym source and the constraint is invisible from inside it.What changed since the last review
The
fastapi_operation_spansunit has been dropped, per @ananthsub's review. It had no effect on Gym's servers in any version Gym allows:fastapi!=0.142.*excludes the one release that double-exports, the lockfile pins 0.135.2 whereFastAPI.__init__has notelemetryparameter at all, and from 0.143 FastAPI stands down when it sees the OTel instrumentor. The measured span volume came from NeMo-RL's pinned Gym, whose server venvs lackopentelemetry-instrumentation-fastapi; NVIDIA-NeMo/RL#4894 is where that belongs. The duplicated gateway helper went with it, leaving only the import guard above.The instance-id docs were also corrected on two points the review raised: a
service.instance.idinOTEL_RESOURCE_ATTRIBUTESis replaced, not kept (lens ranks caller attributes above environment ones, and Gym now always passes one), and the run id prefixes the identity only when the process was configured with one, since lens mintsnemo.run.idlater insidesetup_telemetry.Validation
Run against the final diff, rebased onto
mainat 80cc55d:The 3 failures (
test_connection_pool.py::test_multi_destination_waits_are_attributed_to_the_binding_limitand both tests intest_memory.py) reproduce identically on a cleanorigin/mainworktree and are unrelated to this change. They appear to be macOS-only, which is why the review did not see them;test_call_sites.py::test_supervisor_is_the_only_writer_of_the_active_servers_gauge, listed as failing previously, now passes onmain.Rollouts
N/A — no environment or agent behavior changes. The change affects only a resource attribute; request handling, rollout execution, and scoring paths are untouched.
Compatibility and benchmark impact
No migration required and no benchmark-result effects.
service.instance.idis newly populated where it was previously derived by lens. Dashboards or queries that relied on the old shared"<run id>-rank0"value to group an entire fleet will now see one series per server process; grouping bynemo.run.idgives the previous behavior.A deployment that sets
service.instance.idinOTEL_RESOURCE_ATTRIBUTESno longer keeps that value, because Gym's servers are spawned into one shared environment and a single inherited id would name all of them identically. Passing a resource attribute programmatically still takes precedence.Checklist
pre-commit run --all-files) (so CI lint/format checks pass).git commit -s) (so the DCO check passes).