Skip to content

refactor(llm): move request trace, sinks and recorders into dynamo-telemetry - #220

Draft
ishandhanani wants to merge 2 commits into
idhanani/dyno-221-zmq-helperfrom
idhanani/dyno-224-telemetry
Draft

ishandhanani wants to merge 2 commits into
idhanani/dyno-221-zmq-helperfrom
idhanani/dyno-224-telemetry

Conversation

@ishandhanani

@ishandhanani ishandhanani commented Nov 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

Layer 7 of the lib/llm split stack (DYNO-224). Base: #219 (ZMQ socket helpers in dynamo-runtime), so this PR shows only this layer. It moves everything in dynamo-llm that writes trace or metric records to sinks into a new workspace crate, dynamo-telemetry, at lib/telemetry.

What moved

  • request_trace/ (record types, DYN_REQUEST_TRACE config and policy, the in-process bus, the JSONL, S3, NATS and OTLP sinks, payload capture, agent-context and tool-event records, the tool relay, replay hashing), fpm_trace/ (persistence of forward-pass metrics), recorder.rs (stream-to-JSONL recorder) and perf.rs with perf/ (response recording and logprob analysis) keep their module names: dynamo_llm::request_trace::X is now dynamo_telemetry::request_trace::X, likewise fpm_trace, recorder and perf.
  • The old telemetry module is the new crate root: jsonl_gz, jsonl, stream, bus and parse_sink_names live at dynamo_telemetry::.
  • The integration tests logprob_analysis_integration and request_trace_nats_integration use nothing from dynamo-llm and move to lib/telemetry/tests. The bench request_trace_finish_metadata drives OpenAIPreprocessor, so it stays in lib/llm/benches behind dynamo-llm's request-trace-bench feature.
  • 28 files, 15,092 lines (26 Rust files, plus request_trace/AGENTS.md and its CLAUDE.md), moved with git mv; 16 are byte-identical. Inside the other 12 only this changes: crate::telemetry:: becomes crate:: (9 lines), the new lib.rs gains a crate doc and four module declarations, one logprob test reads its replay from ../llm/tests/data, the two integration tests import dynamo_telemetry, two doc comments drop the telemetry:: prefix, and request_trace/AGENTS.md names the new crate and its commands. lib/llm/src goes from 271,564 to 257,293 lines of Rust (369 to 345 files).
  • No compatibility re-exports: dynamo-llm has no pub use dynamo_telemetry.

Commits

One in-place cut inside dynamo-llm, which builds and passes on its own, then one move commit.

Obstacle Resolution Alternative rejected
Upward imports from the moved modules into dynamo-llm None left to cut. A script that lists every crate:: path of the moved modules, grouped use crate::{...} and test-only ones included, finds only request_trace, telemetry, perf and recorder, which all move. Everything else they need already comes from dynamo-frontend-protocols, dynamo-model-card (LocalModel), dynamo-runtime (with the ZMQ helper from #219), dynamo-worker-selection (the router selection trace record), dynamo-kv-hashing, dynamo-tokens and the published dynamo-protocols n/a
43 pub(crate) declarations in the moved modules, some used by the preprocessor, the HTTP services, the router, the entrypoint and the FPM relay 15 items become pub: exactly the ones a caller outside the moved crate failed to reach when compiled. request_trace: the payload_stream module, X_REQUEST_ID_CONTEXT_KEY, start_tool_event_ingest_from_policy, record_llm_metric_tokens, record_backend_finish_reason_metadata, build_request_end_trace_state and the RequestEndTraceState it returns, finish_reason_metadata_handle, wrap_chat_request_end_stream, wrap_completion_request_end_stream, emit_router_selection, payload::http_header_capture_active. fpm_trace: FpmTrace, FpmTrace::publish_payload, init_from_env_with_shutdown. The other 27 declarations stay crate-private Making everything pub
Cargo features request-trace-s3 and request-trace-bench gate moved code Both move to dynamo-telemetry, and dynamo-llm forwards them, so dynamo-llm/request-trace-s3 in the Python bindings, the wheel build flags and the bench's required-features keep working unchanged. In the new crate request-trace-s3 also enables the optional object_store (feature aws) and uuid dependencies, which only the S3 sink uses An unconditional object_store dependency in the new crate, which code outside the S3 sink does not use
Test data and test helpers No test in the moved modules uses a dynamo-llm helper, and no dynamo-llm test uses a test-only item of theirs, so the new crate needs no testing feature. The one replay file a logprob test reads stays in lib/llm/tests/data and the test points at it, as dynamo-model-card and dynamo-frontend-protocols do Copying the fixture

Callers

  • 45 references in 10 source files of lib/llm/src (preprocessor.rs, http/service/{openai,anthropic,metadata,metrics}.rs, entrypoint/input.rs, entrypoint/input/http.rs, kv_router.rs, kv_router/embedded.rs, fpm_publisher.rs) and the bench now name dynamo_telemetry. Outside lib/llm one file does: the KVBM bindings' leader.rs, which uses recorder::Recorder. lib/bindings/python/rust/llm/fpm.rs references none of the moved modules and is unchanged. dynamo-llm and lib/bindings/kvbm gain the dependency.
  • dynamo-llm drops opentelemetry, opentelemetry_sdk, opentelemetry-otlp, object_store and the dev-dependency approx, which only the moved code used. It keeps flate2 (the RDMA media path), bytemuck (tokens.rs), rmp-serde, async-nats and uuid.
  • areas.yaml gives every moved file the owners it has today: lib/telemetry/ goes to runtime (it owned fpm_trace, recorder, perf and the tests through lib/llm/), the five files of the old telemetry module are listed under observability, and request_trace/ stays shared by runtime, agents and observability. CODEOWNERS is regenerated.
  • Two Python comments that cite the moved paths are updated. No CI path filter lists crate directories that include these modules (filters.yaml matches lib/**), and no docs page cited a moved path; the dynamo_llm split entry in the v1.7.0 section of the deprecations page lists this layer's module paths and log targets. The request-traces page still says the S3 sink needs dynamo-llm built with request-trace-s3, which remains true through the forwarded feature.
  • lib/telemetry/AGENTS.md (with its CLAUDE.md twin) states what the crate owns, what it must not depend on, and how to check it.

Dependency graph

  • cargo tree -p dynamo-telemetry | grep -c dynamo-llm is 0. dynamo-memory, nixl-sys and cudarc do not appear in its tree. object_store does, with or without request-trace-s3: dynamo-lora brings it in through dynamo-model-card and dynamo-frontend-protocols.
  • Direct Dynamo dependencies: dynamo-runtime, dynamo-frontend-protocols, dynamo-model-card, dynamo-worker-selection, dynamo-kv-hashing, dynamo-tokens, and the published dynamo-protocols.
  • The OpenTelemetry span and logging setup in dynamo-runtime is not touched.

Behaviour

  • No message, serialized field, metric, env var or sink output changes. The 15 log statements with an explicit target keep the string dynamo_llm::request_trace, and the error that names dynamo-llm when the S3 sink is not compiled in is unchanged.
  • One difference that follows from any move between crates: the 76 log statements in the moved code without an explicit target now report the module path dynamo_telemetry::... where they reported dynamo_llm::..., as the statements moved by the earlier layers do. A log filter written against dynamo_llm::fpm_trace or dynamo_llm::recorder has to name the new path; no filter in this repository does.

Validation

Cargo commands ran on a remote builder with the repository toolchain (1.96.1).

  • Baseline at 3ffce0e, cargo test -p dynamo-llm -p dynamo-frontend-protocols -p dynamo-model-card --no-fail-fast: 51 binaries, 4555 passed, 0 failed, 31 ignored.
  • Head, same command plus -p dynamo-telemetry: 53 binaries, 4555 passed, 0 failed, 31 ignored. The two extra binaries are the new crate's unit and doc tests. dynamo-llm unit tests go from 2632 passed and 5 ignored to 2455 and 4; the new crate has 177 and 1.
  • Test accounting from cargo test -- --list: all 4586 baseline names exist at head; 4408 keep binary name and path (the 8 tests of the two moved integration binaries among them), 178 moved from dynamo-llm's unit tests to the same leaf name in dynamo-telemetry's; none is missing and none is new.
  • After the cut commit: cargo check -p dynamo-llm --all-targets, cargo clippy --no-deps -p dynamo-llm --all-targets -- -D warnings and cargo test -p dynamo-llm --no-fail-fast pass (45 binaries, 3241 passed, 0 failed, 21 ignored, the same per-binary counts as the baseline).
  • Head: cargo check --workspace --all-targets, cargo clippy --no-deps --workspace --all-targets -- -D warnings, cargo fmt --all -- --check, and cargo check --locked in lib/bindings/python (also with --features request-trace-s3) pass.
  • cargo check -p dynamo-telemetry --all-targets passes alone, with --features request-trace-s3, and with both features under clippy -D warnings. cargo test -p dynamo-telemetry passes alone (177 passed, 1 ignored in the unit tests; 6 passed and 2 ignored in the two integration binaries), and the 12 S3 sink tests, which only request-trace-s3 compiles, pass with the feature.
  • Feature forwarding: cargo check -p dynamo-llm --features request-trace-s3 and cargo check -p dynamo-llm --lib --features request-trace-bench pass.
  • The bench does not compile, at baseline and at head alike: cargo check -p dynamo-llm --bench request_trace_finish_metadata --features request-trace-bench stops on one error, E0063 at benches/request_trace_finish_metadata.rs:80, where the initializer of NvCreateChatCompletionRequest lacks preserve_thinking, responses_tools, thinking and one more field that the struct already has on main. It is the only error at head, so the bench's import of dynamo_telemetry and its calls to the bench hooks resolve. This PR does not fix it.
  • cargo machete at the root, lib/bindings/python, lib/runtime/examples and lib/telemetry reports no unused dependency. cargo metadata --locked passes for the four lock files (root, lib/bindings/python, lib/bindings/kvbm, lib/runtime/examples).
  • pre-commit run --from-ref idhanani/dyno-221-zmq-helper --to-ref HEAD passes, and build_codeowners.py --strict passes with CODEOWNERS matching emit_codeowners.py.
  • cargo check --locked in lib/bindings/kvbm passes, which compiles the one caller outside lib/llm.
  • Not run: the two request_trace_nats_integration tests are ignored by default and need NATS and etcd, at baseline and at head alike. The bench was not run.
  • Both commits are scripted, and re-running each script on its commit's parent reproduces the commit's tree exactly (e3ea6d6d88 for the cut, 232bba4f1b for the move).

Closes DYNO-224

Step 1 of DYNO-224 (layer 7 of splitting lib/llm into smaller crates). request_trace and fpm_trace are about to move into their own crate below dynamo-llm. The preprocessor, the HTTP services, the router, the entrypoint and the FPM relay call 15 of their items that are `pub(crate)` today, which stops working across a crate boundary.

Those 15 become `pub`, and nothing else: the list comes from compiling the moved crate and widening only what a caller outside it failed to reach.

- request_trace: the `payload_stream` module, `X_REQUEST_ID_CONTEXT_KEY`, `start_tool_event_ingest_from_policy`, `record_llm_metric_tokens`, `record_backend_finish_reason_metadata`, `build_request_end_trace_state` with the `RequestEndTraceState` it returns, `finish_reason_metadata_handle`, `wrap_chat_request_end_stream`, `wrap_completion_request_end_stream`, `emit_router_selection` and `payload::http_header_capture_active`. The re-exports in request_trace/mod.rs are split so the rest stays `pub(crate)`.
- fpm_trace: `FpmTrace`, `FpmTrace::publish_payload` and `init_from_env_with_shutdown`.

Inside dynamo-llm the change has no effect. The remaining 27 `pub(crate)` declarations in the modules that move stay crate-private.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
…lemetry

Step 2 of DYNO-224 and layer 7 of splitting lib/llm into smaller crates. Everything in dynamo-llm that writes trace or metric records to sinks moves into a new workspace crate, dynamo-telemetry at lib/telemetry, below dynamo-llm:

- lib/llm/src/request_trace (record types, policy, the in-process bus, the JSONL, S3, NATS and OTLP sinks, payload capture, agent-context and tool-event records, the tool relay, replay hashing), fpm_trace (forward-pass metrics persistence), recorder.rs (stream-to-JSONL recorder) and perf (response recording and logprob analysis) keep their module names: `dynamo_llm::request_trace::X` is now `dynamo_telemetry::request_trace::X`, likewise fpm_trace, recorder and perf
- lib/llm/src/telemetry (the rotating gzip JSONL writer, the plain JSONL writer, the bus, the stream helpers and parse_sink_names) becomes the crate root: `dynamo_llm::telemetry::jsonl_gz` is now `dynamo_telemetry::jsonl_gz`
- the integration tests logprob_analysis_integration and request_trace_nats_integration use nothing from dynamo-llm and move to lib/telemetry/tests. The request_trace_finish_metadata bench drives the preprocessor and stays in lib/llm/benches

The 28 files move with `git mv`, unchanged apart from import paths (`crate::telemetry::` becomes `crate::`), the module declarations and crate doc in the new lib.rs, the replay path of one logprob test, which now points at lib/llm/tests/data, and the crate name in request_trace/AGENTS.md. No upward import had to be cut: these modules already import only the protocol, model-card, runtime, worker-selection, kv-hashing and tokens crates. Explicit log targets keep the string `dynamo_llm::request_trace`, and the message that names dynamo-llm when the S3 sink is not compiled in is unchanged; log statements without an explicit target now carry the `dynamo_telemetry::` module path.

dynamo-llm keeps no re-exports: the preprocessor, the HTTP services, the entrypoint, the router's selection trace, the FPM relay, the bench and the KVBM bindings import dynamo_telemetry directly. The features request-trace-s3 and request-trace-bench move with the code they gate, and dynamo-llm forwards both, so `dynamo-llm/request-trace-s3` in the Python bindings and the bench's required feature keep working. In the new crate request-trace-s3 also turns on the optional object_store and uuid dependencies that only the S3 sink uses. dynamo-llm drops opentelemetry, opentelemetry_sdk, opentelemetry-otlp, object_store and the dev-dependency approx, which only the moved code used.

The new crate depends on dynamo-runtime, dynamo-frontend-protocols, dynamo-model-card, dynamo-worker-selection, dynamo-kv-hashing, dynamo-tokens and the published dynamo-protocols. It does not depend on dynamo-llm. The OpenTelemetry span and logging setup in dynamo-runtime is not touched.

Codeowners gives every moved file the owners it has today: the crate root, fpm_trace, recorder, perf and the tests go to runtime, the five files of the old telemetry module to observability, and request_trace stays shared by runtime, agents and observability. Two Python comments that cite the moved paths are updated.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>
@ishandhanani
ishandhanani force-pushed the idhanani/dyno-224-telemetry branch from f853ab2 to b269955 Compare November 16, 2026 20:34
@ishandhanani
ishandhanani force-pushed the idhanani/dyno-221-zmq-helper branch from 963805f to 702911c Compare November 16, 2026 20:34

This branch had an error being deployed

1 failed deployment
external_collaborator — b2699552 Deployed Nov 16, 2026 by ishandhanani via ok-to-test #744
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant