Repository navigation
Enhance metrics with scheduler, cache, and decoding improvements - #2186
Conversation
🏷️ CI GuideRuns automatically on every eligible PR before approval:
Heavy model tests:
|
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core runtime scheduling/metrics behavior, API SSE timing/logging, and the CI report/query layer simultaneously, requiring careful end-to-end validation in real deployments.
Pull request overview
This PR expands ATOM’s observability pipeline end-to-end (engine → API /metrics → Prometheus → offline HTML report) by adding scheduler-, cache-, workload-, and GPU-forward-focused metrics, while refactoring the OpenAI server metrics exporter into a “compose metrics from owners” model to keep definitions close to their producing components.
Changes:
- Add scheduler-owned cumulative histograms and gauges for queue time, decode batch size, PD KV wait, KV block states, and workload token histograms; propagate per-rank snapshots through the existing metrics snapshot path.
- Add non-blocking GPU device-event forward timing with bounded buffering, plus per-request aggregated prefill GPU forward time (chunk-aware) exported per worker.
- Update the metrics exporter and report generator to support explicit component registration, multi-instance breakdowns, cache tier coverage semantics, and improved SSE parsing/TTFT handling.
File summaries
| File | Description |
|---|---|
| tests/test_scheduler.py | Updates scheduler cache-hit-rate wording to match new semantics. |
| tests/test_scheduler_metrics.py | Adds lifecycle tests covering scheduler queue/KV wait timing, histograms, and snapshot/export invariants. |
| tests/test_observability_report.py | Extends report query and validation tests (units, gauges, grouped queries, timestamps, cache/block counts). |
| tests/test_histogram.py | Adds concurrency and correctness tests for weighted histogram exposition. |
| tests/test_gpu_metrics.py | Adds tests for nonblocking GPU timing, request aggregation across chunks, and worker snapshot routing. |
| tests/test_gc_utils.py | Adjusts GC scrape-cost wording to reference the new collector class. |
| tests/test_ci_latency_reports.py | Expands CI report fixtures to cover new panels, instance breakdown, cache tiers, and KV block counts. |
| tests/entrypoints/test_streaming_dispatch.py | Switches ITL tests to the new metrics composition API and shared bucket definitions. |
| tests/entrypoints/test_stream_silence_watchdog.py | Updates logging tests to exercise the real client-stream boundary behavior. |
| tests/entrypoints/test_request_timing.py | Reworks TTFT tests around middleware eligibility + stream-level parsing; adds more SSE shapes. |
| tests/entrypoints/test_metrics.py | Refactors exporter tests for snapshot pinning, component registration, registry isolation, and concurrent scrapes. |
| docs/agentic_metrics.md | Documents new metrics definitions, boundaries, tiers, and report semantics (incl. multi-instance). |
| atom/utils/histogram.py | Introduces cumulative and weighted histogram helpers shared by metrics owners. |
| atom/utils/gc_utils.py | Adds GCMetricsCollector for O(1) live GC scrape metrics. |
| atom/model_engine/sequence.py | Adds per-sequence state used to track initial prefill GPU chunk timing boundaries. |
| atom/model_engine/scheduler.py | Integrates scheduler metrics (enqueue, KV wait lifecycle, forward recording) and freezes prefill chunk boundaries. |
| atom/model_engine/scheduler_metrics.py | Adds scheduler cumulative metric collection + Prometheus exposition adapter. |
| atom/model_engine/pp_engine_core.py | Pushes periodic metrics from PP procs; records scheduler forward observations at PP head only. |
| atom/model_engine/model_runner.py | Adds GPUForwardMetrics installation, worker snapshot collection, and forward timing decoration. |
| atom/model_engine/llm_engine.py | Aggregates per-rank scheduler/worker metrics and preserves cache tier semantics for missing vs zero. |
| atom/model_engine/gpu_metrics.py | Implements bounded, nonblocking device-event timing + per-request prefill aggregation + Prometheus exposition adapter. |
| atom/model_engine/engine_utility.py | Pushes worker forward snapshots and augments scheduler snapshot with queue/KV-state semantics. |
| atom/model_engine/engine_stats.py | Adds Prometheus exposition adapter for EngineStats + cache tier optionality handling. |
| atom/model_engine/engine_core.py | Stamps request receipt time earlier, records scheduler forward observations, and increases metrics push cadence. |
| atom/model_engine/dp_metrics.py | Adds Prometheus exposition adapter for DP router statistics. |
| atom/model_engine/async_proc.py | Routes worker forward-metrics snapshots out-of-band without polluting KV/forward result queues. |
| atom/kv_transfer/offload/metrics.py | Adds Prometheus exposition adapter for offload connector cumulative statistics. |
| atom/entrypoints/openai/streaming_dispatch.py | Adds StreamMetrics (ITL + silence gauge) with shared ITL bucket constants. |
| atom/entrypoints/openai/sse.py | Adds SSE frame inspection helper for parsing coalesced/complete local frames. |
| atom/entrypoints/openai/request_timing.py | Moves generated-output detection into a reusable helper and adds RequestMetrics for TTFT instrumentation. |
| atom/entrypoints/openai/metrics.py | Refactors exporter into a snapshot cache + explicit registry composition model with scrape snapshot pinning. |
| atom/entrypoints/openai/metrics_setup.py | Adds a single composition point to register component collectors and local instruments per API instance. |
| atom/entrypoints/openai/api_server.py | Switches to composed metrics, unifies stream parsing for logging+TTFT, and updates refresh cadence. |
| .github/scripts/atomesh/observability/README.md | Updates report documentation to cover new panels, units, instance selectors, and cache tier semantics. |
| .github/scripts/atomesh/observability/export_report.py | Expands panels, adds grouped-by-instance query support, cache/block count series, and stricter validation. |
| .github/scripts/atomesh/observability/collect_metrics.py | Adjusts scrape cadence and report metadata/empty-report generation to match new panel/statistic model. |
Review details
- Files reviewed: 37/37 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
There was a problem hiding this comment.
🔵 Needs a closer look
It introduces cross-cutting observability changes spanning engine scheduling, worker telemetry routing, API streaming parsing, and CI report generation that warrant a final human validation pass for runtime/operational impact.
Review details
- Files reviewed: 32/32 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It makes broad, coupled changes across the scheduler/engine loops, API streaming instrumentation, and Prometheus/reporting pipeline that warrant a human review for runtime overhead, compatibility, and operational rollout risk.
Review details
- Files reviewed: 32/32 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It makes cross-cutting changes to scheduling/engine loops, OpenAI streaming parsing, and introduces new 1s-cadence metrics plus optional GPU event timing that warrant manual validation and performance/regression review.
Review details
- Files reviewed: 36/36 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It changes core engine scheduling loops, streaming response parsing, and the metrics/reporting pipeline broadly, so it needs a final human review with real workload verification (perf + correctness) before approval.
Review details
- Files reviewed: 36/36 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
It touches core scheduling/engine loops, streaming response handling, and CI observability/reporting in a performance-sensitive path, so it warrants final human review despite strong test coverage.
Review details
- Files reviewed: 38/38 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
A confirmed resource-leak bug exists in GPUForwardMetrics.measure() where event pairs are not returned to the free pool on exceptions, risking unbounded CUDA/HIP event allocation under repeated failures.
Review details
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
atom/model_engine/gpu_metrics.py:115
- In the exception path inside
GPUForwardMetrics.measure(), the allocated event pair is neither appended topendingnor returned tofree. If a forward raises repeatedly, this will keep allocating new CUDA/HIP events (until GC), increasing memory/handle pressure unnecessarily even though no samples are published for failures.
- Files reviewed: 41/41 changed files
- Comments generated: 0 new
- Review effort level: Lite
f1290ea to
78fba08
Compare
There was a problem hiding this comment.
🟡 Changes recommended
WeightedHistogram._child_samples() does not keep its lock held while iterating samples, which can allow scrapes to interleave with observations and read inconsistent histogram state.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 41/41 changed files
- Comments generated: 2
- Review effort level: Lite
…ters Record per-request and per-forward prefill work, decode context totals, and nonblocking GPU forward timings. Report GPU and LMCache reuse together, include pure-PD consumer cache accounting, and add independent service-instance filters.
There was a problem hiding this comment.
🔵 Needs a closer look
Five moderate findings remain unresolved across DP readiness, remote metrics export, process initialization, metric labeling, and cache accounting.
Review details
Suppressed comments (5)
.github/scripts/atomesh/pd_server_atom.sh:518
- For a multi-node DP service,
api_server.maintakes thedp_rank > 0branch and exposes only/metricson remote workers, so this probe can never succeed against those nodes. The coordinator paths below still pass everyprefill_ips/decode_ipsentry to this helper, which can make startup time out before the router and benchmark run; keep all worker addresses for Prometheus scraping, but probe only the coordinator API for each DP group (or skip remote DP entries).
until python3 "${ATOMESH_SCRIPT_DIR}/../../../scripts/check_server_ready.py" "${url}" >/dev/null; do
atom/entrypoints/metrics.py:23
- The worker-node metrics server registers only
MultiProcessCollector, but the queue-state, KV-block, cache, and legacy engine metrics are materialized by_AtomMetricsCollectorfrom engine snapshots, not written to multiprocess files. Consequently the remote DP endpoint used by the new per-instance report has noatom:scheduler_requests,atom:scheduler_kv_cache_blocks, or cache/state series, leaving those panels incomplete for remote instances. Export these state metrics natively or provide/update an equivalent snapshot collector on each worker.
registry = CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
return start_http_server(port, addr=host, registry=registry)
atom/entrypoints/openai/api_server.py:387
openai_server.pyis also re-executed as the multiprocessing spawn main module, so every EngineCore child reaches this top-level API initialization even though it never serves HTTP. In multiprocess mode this createsRequestMetrics/StreamMetricsmmap files (including zero-valued API histogram children) in every engine process, and the API'sMultiProcessCollectorscans them on each scrape. Keep API-only instruments out of spawned engine processes by guarding the API import/initialization to the serving process; otherwise scrape overhead and native metric storage scale with all engine workers.
_metrics_exporter, _request_metrics, _stream_metrics = create_metrics_exporter()
atom/model_engine/scheduler.py:520
- The first
SchedulerMetricsargument is named and exported asdp_rank, but this expression foldspipeline_parallel_rankinto it. For a DP=0, PP=1 worker the scheduler series therefore saysdp_rank="1", while the GPU series for the same worker saysdp_rank="0", pp_rank="1"; grouping or correlating metrics bydp_rankmisattributes the stage. Preserve the actual DP rank and add a separatepp_ranklabel (or rename this label to an explicitly flattened worker rank).
self.waiting: deque[Sequence] = deque()
self.running: deque[Sequence] = deque()
self.config = config
pc = getattr(config, "parallel_config", None)
self.metrics = SchedulerMetrics(
getattr(pc, "data_parallel_rank", 0)
atom/model_engine/scheduler.py:2207
_schedule_prefill_seqis called for every chunk/re-admission (including the chunked-prefill paths above), so this incrementsEngineStatscache totals once per chunk rather than once per request. A long prompt will therefore overcount requests, full-token denominators, and reuse; guard this admission with a request-level marker (and preserve the marker across preemption) or record it at the one-time cache-admission transition instead.
num_batched_tokens: int,
- Files reviewed: 50/50 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved findings affect coordinator-only readiness probes, per-node snapshot metrics, KV block accounting, GPU shutdown draining, and service-role labeling.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (6)
.github/scripts/atomesh/pd_server_atom.sh:1363
- In the ordinary multi-node layout,
prefill_ipscontains one address per prefill node, while only the coordinator serves/v1/models; the other DP processes take the metrics-only branch. This loop therefore waits for a model list that remote workers never expose and can time out before the router starts. Wait only on the prefill coordinator; retain all node addresses as metrics targets.
wait_api_ready "http://${prefill_ips[$idx]}:${prefill_ports[$idx]}/v1/models" \
"prefill-${prefill_ips[$idx]}:${prefill_ports[$idx]}" \
"${WAIT_SERVER_TIMEOUT}" "${server_pid}"
.github/scripts/atomesh/pd_server_atom.sh:1368
- The decode address list in the ordinary multi-node layout also includes every DP node, but non-coordinator decode workers expose metrics only. This readiness loop consequently waits for
/v1/modelson endpoints that cannot answer it and blocks startup. Restrict the readiness probe to the decode coordinator while continuing to scrape every address for metrics.
wait_api_ready "http://${decode_ips[$idx]}:${decode_ports[$idx]}/v1/models" \
"decode-${decode_ips[$idx]}:${decode_ports[$idx]}" \
"${WAIT_SERVER_TIMEOUT}"
atom/entrypoints/metrics.py:18
- This endpoint is used for every non-coordinator DP target, but it exposes only the native multiprocess collector. The report's queue/block/cache families (
atom:scheduler_requests,atom:scheduler_kv_cache_blocks, andatom:prefix_cache_*) are emitted by the snapshot collector inopenai/metrics.py, not by the worker instruments. As a result, scraping the added remote targets omits their snapshot state and undercounts multi-node cache reuse and utilization. Publish those snapshot families on each node or keep them out of the per-node aggregation.
"""Expose this node's native instruments, without coordinator snapshots."""
atom/model_engine/block_pool.py:154
- The pool's own invariant says these containers can retain stale IDs and that
_freemembership determines whether a block is actually free. Counting_cachedand_protecteddirectly therefore overcounts evictable blocks after claim/retirement churn (and can make the derived vacant count negative), so the exported KV partition becomes incorrect. Keep the free-set/hash filter here.
return len(self._cached) + len(self._protected)
atom/model_engine/engine_utility.py:340
call_func("poll_forward_metrics")is fire-and-forget, so shutdown can terminate the runner after this request has been queued but before a final device-event poll observes events that complete after the last periodic tick. The final Prometheus scrape can consequently omit the last GPU samples (and incomplete request-prefill timings); add a shutdown drain/final poll that waits for the RPC to execute and gives pending events a bounded final observation window.
# Poll ready device events even after the final forward. Workers write
# native metrics directly; this RPC has no response payload.
if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
self.runner_mgr.call_func("poll_forward_metrics")
atom/model_engine/model_runner.py:833
engine_roleis set to"default"unlessconfig.enable_rapidserveis true. The PD launcher starts its services with--kv-transfer-configbut does not enable--enable-rapidserveby default, so an ordinary agentic PD run records GPU histograms underrole="default"; the report queriesrole="prefill"androle="decode"and therefore shows empty GPU panels despite device timing being enabled. Author must derive/propagate the actual prefill/decode service role independently of the RapidServe feature flag.
engine_role=(
("decode" if config.disagg_is_decode else "prefill")
if config.enable_rapidserve
else "default"
),
- Files reviewed: 50/50 changed files
- Comments generated: 1
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical metric-series and rank-labeling defects remain, alongside moderate reporting, readiness, metrics-lifecycle, and shutdown issues.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (8)
.github/scripts/atomesh/observability/collect_metrics.py:262
finalize_reportonly inspects the aggregaterecords/seriesfields and ignoresp["instances"]. If an aggregate query fails while the grouped per-instance queries succeed, this adds a false “No samples” error (and can mark the run partial/unavailable) even though the report has valid data. Include instance bundles in the sample check.
if not p.get("records")
and not any(
any(v is not None for _, v in points) for points in p["series"].values()
)
.github/scripts/atomesh/observability/export_report.py:149
- [verified] This always adds a
prefill_kv_blockspanel, but the real disaggregated prefill path usesPrefillScheduler, whoseblock_manageris explicitlyNone;EngineUtilityHandler.collect_metrics()therefore emits nokv_blocks_*state for that role. The PromQL query will return no samples and this panel is permanently empty for every normal P/D deployment. Author must either omit the prefill block panel or provide a real supported source for prefill block state.
**common,
"id": f"{role}_kv_blocks",
"title": f"{role.title()} KV block utilization",
"detail": "Used, evictable cached, and vacant blocks · capacity-weighted across pools",
"metric": "atom:scheduler_kv_cache_blocks",
"unit": "%",
"kind": "blocks",
.github/scripts/atomesh/pd_server_atom.sh:1363
- The
prefill_ips/decode_ipsarrays are also used as all-node metrics targets, and in a multi-node DP deployment can include non-coordinator ranks. Those ranks take the newdp_rank > 0path and serve only/metrics, so probing/v1/modelshere can never succeed and rank-0 startup times out. Readiness should probe only each service coordinator; keep every address for metrics collection.
wait_api_ready "http://${prefill_ips[$idx]}:${prefill_ports[$idx]}/v1/models" \
"prefill-${prefill_ips[$idx]}:${prefill_ports[$idx]}" \
"${WAIT_SERVER_TIMEOUT}" "${server_pid}"
atom/entrypoints/metrics.py:23
- This endpoint is used by the
dp_rank > 0startup path, but it registers onlyMultiProcessCollector. The existing queue, KV/cache, and offload metrics are emitted byAtomMetricsExporterfromLLMEnginesnapshots, and that remote path never starts the snapshot refresh/lifespan. In a multi-node DP run, scraping these advertised remote targets therefore omits those panels and undercounts cache reuse; expose the snapshot exporter/refresh on each node or aggregate those snapshots into the coordinator before serving.
registry = CollectorRegistry()
multiprocess.MultiProcessCollector(registry)
return start_http_server(port, addr=host, registry=registry)
atom/entrypoints/openai/api_server.py:2960
create_engine()constructs and returns anLLMEngineafter its cores become ready; it does not block for the worker's lifetime. Therefore thisfinallyruns immediately, shutting down the only metrics HTTP server, while the returned engine is discarded (and its child processes are not owned by a live reference). DP ranks greater than zero will stop exposing metrics as soon as startup completes. Keep the engine/server alive until the worker terminates, and close both during shutdown.
engine_args.create_engine(tokenizer=tokenizer)
finally:
server.shutdown()
server.server_close()
atom/metrics/scheduler.py:118
- The decode counterpart has the same unbounded per-request cardinality: every
request_id/sequence_id/started_attuple remains in native multiprocess storage for the entire service lifetime. This can make the default/metricsendpoint and its backing files grow without bound; use bounded/opt-in request telemetry or an expiry path here as well.
self.decode_request_context_tokens = Gauge(
"atom:decode_request_context_tokens",
"Logical context length at first real decode dispatch, once per request sequence; retained until metrics storage is cleaned.",
[*labels, "request_id", "sequence_id", "started_at"],
multiprocess_mode="max",
registry=registry,
atom/model_engine/engine_core.py:346
- If the shutdown request is consumed immediately after the last forward, this loop exits before another scheduled
push_metricscall. Device events from that final forward may therefore never be polled or exported before the worker is destroyed, so the report's final scrape can silently miss the tail of GPU timing data. Add a shutdown-time poll/drain or otherwise keep collection alive until pending observations settle.
now = time.monotonic()
if now >= next_metrics_push:
next_metrics_push = now + metrics_interval
self.utility_handler.push_metrics()
atom/model_engine/scheduler.py:3471
- This duplicate constructor applies the same PP offset to the
dp_ranklabel, so disaggregated prefill scheduler metrics are mislabelled under PP. Keep this label consistent with the data-parallel rank used by the engine-core manager; pipeline stage identity is already a separate label for GPU metrics.
self.metrics = SchedulerMetrics(
getattr(pc, "data_parallel_rank", 0)
* getattr(config, "pipeline_parallel_size", 1)
+ getattr(pc, "pipeline_parallel_rank", 0),
self._METRICS_ROLE or "default",
- Files reviewed: 51/51 changed files
- Comments generated: 2
- Review effort level: Lite
There was a problem hiding this comment.
🔵 Needs a closer look
Three moderate findings remain unresolved, along with one documentation nit.
Review details
Suppressed comments (4)
atom/model_engine/engine_utility.py:340
- If shutdown is consumed before the next periodic
push_metrics()tick, the worker exits without anotherpoll_forward_metricscall. CUDA events from the last forward can therefore remain pending and the final scrape/report silently omits the last GPU step (and possibly the completed request-prefill sample), even though the service has stopped cleanly. Add a bounded worker-side drain/poll during engine shutdown before the metrics directory is collected.
# Poll ready device events even after the final forward. Workers write
# native metrics directly; this RPC has no response payload.
if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
self.runner_mgr.call_func("poll_forward_metrics")
atom/model_engine/scheduler.py:3470
- [verified] The same composite-rank label is used by
PrefillScheduler; its snapshots are also keyed by data-parallel rank, so PP+DP prefill scheduler series get the wrongdp_ranklabel for every nonzero DP rank. Author must keep this label equal todata_parallel_rankand leave pipeline rank as a separate dimension if it is needed.
getattr(pc, "data_parallel_rank", 0)
* getattr(config, "pipeline_parallel_size", 1)
+ getattr(pc, "pipeline_parallel_rank", 0),
atom/model_engine/scheduler.py:522
- [verified] This labels the scheduler with a composite
dp_rank(dp * pp_size + pp_rank), but the metrics snapshot is keyed and aggregated by the actual data-parallel rank, and PP downstream stages do not publish scheduler snapshots. In a PP+DP deployment, for example DP rank 1 withpp_size=2is exported asdp_rank="2", so the label no longer identifies the DP rank and can misattribute/skip per-rank scheduler series. Author must use the actual data-parallel rank here (PP is already represented by its own label where applicable).
getattr(pc, "data_parallel_rank", 0)
* getattr(config, "pipeline_parallel_size", 1)
+ getattr(pc, "pipeline_parallel_rank", 0),
docs/metrics_guide.md:271
- This rule contradicts the request-context gauges documented above and implemented in
atom/metrics/scheduler.py, which intentionally userequest_idlabels so completed requests can be reported. Qualify the rule to exempt those explicitly documented per-request gauges; otherwise the metric-addition guidance tells future contributors to remove a required label.
- Files reviewed: 46/46 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
Two critical and five moderate findings remain unresolved, including duplicate Prometheus registrations, missing metric exports, and incomplete GPU-event draining.
Get a fresh assessment by requesting another Copilot review.
Review details
Suppressed comments (5)
atom/entrypoints/openai/metrics_setup.py:26
prometheus_clienttreats an explicitregistry=Noneas “do not register this instrument”; it does not mean “use the default native registry.” In a multiprocess launch this leaves the request/stream histograms out of both the default registry and the mmap files, soMultiProcessCollectorcannot export TTFT or ITL at all. Register these instruments in the native defaultREGISTRY(while keepingexporter.registryfor the snapshot collector) instead of passingNone.
multiprocess.MultiProcessCollector(registry)
# Register only the multiprocess collector, not its instruments too.
registry = None
request_metrics = RequestMetrics(registry)
atom/metrics/gpu.py:38
- This has the same registration problem for GPU telemetry:
ModelRunnerconstructsGPUForwardMetricswithout a registry, and the explicitregistry=Nonesuppresses registration of both histograms. The worker can observe and locally collect these objects, but no native multiprocess file is written, so the GPU panels remain empty in the real service. Register the default instance in the native registry, with an injectable isolated registry for tests.
class GPUForwardMetrics:
def __init__(
self,
event_factory,
max_pending=256,
max_requests=4096,
*,
dp_rank=0,
pp_rank=0,
tp_rank=0,
engine_role="default",
registry=None,
atom/metrics/prometheus.py:18
- This endpoint registers only
MultiProcessCollector, while the newatom:scheduler_requestsandatom:scheduler_kv_cache_blocksfamilies are emitted by the API's snapshot collector fromEngineUtilityHandler.collect_metrics, not by native worker instruments. In thedp_rank > 0startup path this server therefore cannot expose queue/block state (or the other snapshot-only cache/state metrics), so scraping every configured DP instance still produces incomplete per-instance panels. Publish the snapshot collector on remote nodes or explicitly exclude these metrics from those targets.
"""Expose this node's native instruments, without coordinator snapshots."""
atom/metrics/scheduler.py:55
- The scheduler creates
SchedulerMetricswithout a registry, and this default is passed explicitly to everyHistogram/Gauge. Inprometheus_client, that disables registration, so the scheduler histograms and request-context gauges never reach the native multiprocess files or/metrics; the snapshot only contains queue/block state and cannot replace them. Use the process's native default registry (or explicitly inject one) for the production default, while allowing isolated registries in tests.
class SchedulerMetrics:
def __init__(self, dp_rank=0, engine_role="default", *, registry=None):
labels = {"dp_rank": str(dp_rank), "engine_role": engine_role}
atom/model_engine/engine_utility.py:340
- [verified] This only enqueues
poll_forward_metricsthrough the non-blocking RPC (wait_out=False); the engine shutdown paths then stop the workers without a final bounded poll/drain. Any forward whose end event is still incomplete at the last periodic poll is therefore never observed, so the final report can permanently omit its per-step and per-request GPU samples even after the final scrape. Author must add a bounded shutdown drain (or otherwise wait for pending device events) before terminating the workers and cover a forward that completes after the last periodic poll.
# Poll ready device events even after the final forward. Workers write
# native metrics directly; this RPC has no response payload.
if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
self.runner_mgr.call_func("poll_forward_metrics")
- Files reviewed: 45/45 changed files
- Comments generated: 2
- Review effort level: Lite
| pp_rank=0, | ||
| tp_rank=0, | ||
| engine_role="default", | ||
| registry=None, |
| self.running: deque[Sequence] = deque() | ||
| self.config = config | ||
| pc = getattr(config, "parallel_config", None) | ||
| self.metrics = SchedulerMetrics( |
There was a problem hiding this comment.
🔵 Needs a closer look
Five unresolved moderate findings block approval.
Review details
Suppressed comments (5)
.github/scripts/atomesh/observability/export_report.py:123
- [verified] In the real disaggregated PD path,
PrefillSchedulerhas no block manager and does not perform prefix-cache accounting, so it cannot emit the prefill KV-block/cache families. Creating these panels for both roles therefore produces empty prefill panels and causes otherwise successful PD reports to be marked partial; the fixture tests synthesize data that the prefill service does not produce. Author must omit decode-owned panels from the prefill role or source their values from the decode owner.
roles = ("standalone",) if deployment == "standalone" else ("prefill", "decode")
for role in roles:
selector = f'job="atom",role="{role}"'
common = {"selector": selector, "label": f"{role.upper()} · SCHEDULER"}
panels.extend(
.github/scripts/atomesh/observability/export_report.py:366
- [verified] The cache coverage guard compares GPU/LMCache series only with prompt series. If an entire target is missing, all of those series disappear for that instance, so the counts still match and the aggregate percentage is calculated from the surviving instances instead of remaining unknown, contrary to the documented incomplete-instance behavior. Author must include configured/
uptarget coverage in this check or propagate missing coverage.
if panel.get("cache_breakdown") and state in {"gpu", "lmcache"}:
# A partial rollout/missing tier must not silently lower total reuse.
# Zero counters from a service without LMCache remain valid samples.
inputs = increase(metrics["prompt"])
result = f"({result} and ({count}({values}) == {count}({inputs})))"
atom/model_engine/engine_utility.py:340
- [verified]
poll_forward_metricsis invoked only by periodic metric pushes; the shutdown/finalization paths do not issue a final poll or drain before the worker exits. A forward whose device event becomes ready after the last tick remains pending, so the final GPU report can omit its step and completed prefill-request sample. Author must add a bounded final poll/drain before worker shutdown.
# Poll ready device events even after the final forward. Workers write
# native metrics directly; this RPC has no response payload.
if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
self.runner_mgr.call_func("poll_forward_metrics")
atom/model_engine/scheduler.py:523
- [verified] The scheduler histogram label is computed as
data_parallel_rank * pipeline_parallel_size + pipeline_parallel_rank, but metrics snapshots andLLMEngine.get_metrics_statistics()are keyed by the data-parallel rank only; downstream PP stages suppress scheduler snapshots. With DP=2 and PP=2, head observations are labeled 0 and 2 while the corresponding snapshot ranks are 0 and 1, so per-DP histograms are misidentified. Author must use the data-parallel rank here (and add a separate PP label only if stage-level scheduler observations are later emitted).
self.metrics = SchedulerMetrics(
getattr(pc, "data_parallel_rank", 0)
* getattr(config, "pipeline_parallel_size", 1)
+ getattr(pc, "pipeline_parallel_rank", 0),
self._METRICS_ROLE or "default",
atom/model_engine/scheduler.py:3471
- [verified] This second constructor repeats the PP-folded value, so
PrefillScheduler/other scheduler histograms use the same incorrect rank mapping described above. In a PP deployment the native metric labels no longer line up with the rank keys used by the aggregator, which can mis-group per-rank data. Author must pass the data-parallel rank directly here as well.
self.metrics = SchedulerMetrics(
getattr(pc, "data_parallel_rank", 0)
* getattr(config, "pipeline_parallel_size", 1)
+ getattr(pc, "pipeline_parallel_rank", 0),
self._METRICS_ROLE or "default",
- Files reviewed: 45/45 changed files
- Comments generated: 0 new
- Review effort level: Lite
…ded by `Run non-GPU unit tests` has been red on main since ROCm#2186 landed. The two changes are individually green and collide on merge: ROCm#2028 made `Scheduler.preempt` strip the placeholder tokens postprocess appended, reading `seq.num_placeholder_tokens`, and ROCm#2186 added a test that drives `preempt` with a `SimpleNamespace` carrying two fields. strip = seq.num_placeholder_tokens E AttributeError: SimpleNamespace object has no attribute num_placeholder_tokens The double now carries the three the strip width is bounded by, each at the value a real `Sequence` starts it at, which leaves the width at zero and the test asserting what it always asserted. Unrelated to the KV budget change in this branch; it is here because it blocks the branch. Reproduced on main at 709c25b alone -- the scheduled Pre Checkin run 35156585585 failed the same job at the same step with no commit of ours in it. Co-authored-by: Cursor <cursoragent@cursor.com>
…m#2186) * feat(metrics): add agentic scheduler metrics and report filters * feat(metrics): add cache reuse and workload metrics with instance filters Record per-request and per-forward prefill work, decode context totals, and nonblocking GPU forward timings. Report GPU and LMCache reuse together, include pure-PD consumer cache accounting, and add independent service-instance filters. * feat(metrics): add per-request decode context distribution * feat(metrics): sum prefill GPU forward time per request * refactor(metrics): share client stream parsing for logging and TTFT * refactor(metrics): define instruments in owning components Move metric definitions and observation interfaces into request, stream, scheduler, GPU, engine, DP, offload and GC modules. Compose them explicitly through the native Prometheus registry and keep the exporter focused on cached snapshots and refresh health. Pin one snapshot per render and reserve optional family names during registration. Extend the standard Histogram for token-weighted ITL while preserving metric names, labels, buckets, cumulative values and snapshot IPC. Document how to add and register component metrics. Validation in rocm/atom-dev:nightly_202609071454: - Broader regression: 2588 passed, 56 skipped, 3 xfailed. - prometheus-client 0.25.0 focused suite: 180 passed, 1 GPU skip. - Six-scenario exposition parity on 0.23.1 and 0.25.0; full snapshot preserves all 80 metric families and 1202 samples. * refactor(metrics): limit additions to dashboard metrics * docs: remove agentic metrics reference * perf(metrics): make GPU timing opt-in and poll events in FIFO order * feat(metrics): add prefill context length metrics and dashboard panels * docs(metrics): document every exported metric and how to add one Restores, in English, the metric reference and registration how-to that was removed wholesale with docs/agentic_metrics.md in 5cd92d8 -- that delete targeted the agentic framing and took the contributor-facing guide with it, leaving the collector contract described only in a docstring. Covers all 72 exported metric families grouped by owning component and the two ways a number reaches /metrics: an in-process instrument registered against the exporter registry, or a snapshot collector whose engine-side accumulator rides the existing METRICS push and is converted to families at render time. The how-to gives runnable examples for both paths plus the collector contract: collect(None) declares every possible family without I/O, collect(snapshot) only reads, and a missing snapshot field must not render as zero. Indexed from the CLAUDE.md guide table, and the cross-link 5cd92d8 dropped from the CI observability README is restored. * fix(metrics): remove pool scans and correct histogram bounds Maintain reusable free-block counts incrementally, extend batch context buckets, and reserve original collector names during registration. Cover pool lifecycle transitions, large contexts, metric collisions, and deterministic LMCache dependency failure. * fix(metrics): render scrapes off the API event loop * perf(metrics): simplify GPU histograms and recycle timing events Pool prefill, decode and mixed forward steps into one per-worker GPU histogram while retaining independent cumulative prefill-request timing. Keep worker identities, histogram buckets and report percentiles; update HTML titles and document the removal of the phase label. Recycle timing event pairs after failed forwards, preserve exception propagation and exclude failed samples from both distributions. Validate pooled histogram parity, request accumulation, event reuse and Prometheus-to-HTML integration: 98 passed, one real-device test skipped. * fix(metrics): make update and scrape intervals configurable Use one validated ATOM_METRICS_UPDATE_INTERVAL_S setting for ordinary, DP and PP engine metrics pushes and API snapshot refresh, defaulting to one second. Read the shared setting once when each loop starts. Configure Prometheus scraping independently with --scrape-interval-seconds and a shared function/CLI default. Bound scrape timeouts and adapt collector waits and histogram query windows to longer intervals. Document both settings. Validation: 160 focused tests passed, including all engine loop variants, API lifespan, invalid values and real Prometheus collection at 500 ms. * refactor(metrics): derive reusable block count from pool queues * docs(metrics): clarify GPU timer initialization and graph capture * refactor(metrics): use native multiprocess histograms * refactor(metrics): inherit standard histogram observe and collection * fix: validate API model lists before reporting readiness Use a shared model-list probe for local and native ATOM CI startup gates so DP worker metrics responses cannot report readiness. Document coordinator probing for multi-node DP and cover invalid responses and startup transitions. * fix: fall back to default for invalid metrics intervals Warn and use the default one-second interval when ATOM_METRICS_UPDATE_INTERVAL_S is invalid, allowing startup to continue. Preserve valid overrides and cover fallback behavior in environment parsing, all engine loops, and API startup. * fix: expose first decode context once per request via metrics * fix: expose prefill context once per request via metrics * fix: select meaningful gauge panel headlines * fix: avoid retaining shared-cache prefill requests in metrics * fix: time GPU forwards during hidden-state extraction * fix: preserve newer request timing on stale event errors * test: import current checkout in spawned metrics workers * style: fix rollout model runner import grouping * docs: explain metrics-only prefill GPU tracking * refactor: move runtime metrics into atom.metrics * refactor: centralize shared metrics infrastructure * chore: drop readiness probe changes from metrics PR * test: trim redundant metrics regression coverage * test: consolidate TTFT scenarios and drop redundant query count
Motivation
Existing TTFT and ITL metrics provide limited visibility into scheduling delays, cache reuse, workload shape, and GPU execution time. This PR adds these measurements and integrates them into the offline agentic benchmark report to help diagnose latency and throughput
behavior across prefill and decode services.
Technical Details
Scheduler metrics: Add queue time, successful PD KV transfer wait time, actual decode batch size, prefill token counts, and prefill/decode context-length distributions. Expose request queue states and KV block utilization, with block-state counts maintained
incrementally.
GPU timing: Add optional per-worker target-model forward timing and cumulative prefill timing across each request’s initial chunks. Timing uses nonblocking device-event polling, bounded buffering, and event reuse. Request-level measurements include the full duration
of each participating batch; PP measurements cover each local stage.
Cache accounting: Expose supplemental LMCache reuse and record the decode consumer’s local prefix reuse after successful remote KV receipt. Reports display total reuse, GPU reuse, and LMCache reuse against a shared input-token denominator.
Metrics organization: Centralize GPU timing, scheduler instruments, histogram helpers, Prometheus initialization/HTTP export, snapshot collection/rendering, and request/stream instruments in
atom/metrics/. Keep API composition, request middleware, streaming dispatch, andEngineStatswith their owning components. The API injects its live stream-silence callback into the exporter, which reads it on the event loop before rendering in a worker thread. Preserve metric names, buckets, labels, multiprocess initialization order, and observation semantics; update imports, tests, and the metrics guide.Streaming instrumentation: Share SSE parsing between request logging and streaming TTFT detection, while preserving first-generated-output semantics.
Reports and CI: Add workload, cache, queue, and GPU panels; role/category filters; independent prefill/decode instance selectors; and corresponding table/CSV data. Aggregate distributions from histogram buckets and ratios from raw counts. Increase engine refresh and
CI scrape cadence to one second.
GPU timing is disabled by default and can be enabled with ATOM_ENABLE_METRICS_DEVICE_TIMER=1. Agentic CI enables it for both service roles, with environment overrides supported.


[glm-5.2-mxfp4-glm-52-mxfp4-1p1d-cpp4-dcp4-agentic-lmcache-1m-c48-nightly](https://github.com/ROCm/ATOM/actions/runs/34489056541/job/102928136600)
Test Plan
Focused tests cover:
Test Result
rocm/atom-dev:latest. Skips require a GPU or an explicitly configured Prometheus binary. Prometheus bootstrap import-order and package-independence checks also passed.bash -n.engine_core.pyandapi_server.pyare unchanged.rocm/atom-dev:latest(2026-09-15 build, digestsha256:38714402a1b5a408d2a57d8f9d5900afe31b33302375b0981ee34509e36c3773): GLM-5.2 MXFP4, CPP4 prefill + TP4/DCP4 decode, Mooncake, MTP 3, prefix caching, LMCache CPU offload, and GPU timing enabled. The container's Prometheus client was upgraded from 0.23.1 to the PR-required 0.25.0.engine_role="default"; the CI report distinguishes prefill/decode using the scrape target'srolelabel.GPU telemetry is asynchronous. The live test verified settled idle observations; ensuring final observations are included before every report collection stops remains a follow-up.