Skip to content

Enhance metrics with scheduler, cache, and decoding improvements - #2186

Merged
valarLip merged 36 commits into
mainfrom
Jasen/metrics2
Sep 16, 2026
Merged

valarLip merged 36 commits into
mainfrom
Jasen/metrics2

Conversation

@Jasen2201

@Jasen2201 Jasen2201 commented Sep 10, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Existing TTFT and ITL metrics provide limited visibility into scheduling delays, cache reuse, workload shape, and GPU execution time. This PR adds these measurements and integrates them into the offline agentic benchmark report to help diagnose latency and throughput
behavior across prefill and decode services.

Technical Details

  • Scheduler metrics: Add queue time, successful PD KV transfer wait time, actual decode batch size, prefill token counts, and prefill/decode context-length distributions. Expose request queue states and KV block utilization, with block-state counts maintained
    incrementally.

  • GPU timing: Add optional per-worker target-model forward timing and cumulative prefill timing across each request’s initial chunks. Timing uses nonblocking device-event polling, bounded buffering, and event reuse. Request-level measurements include the full duration
    of each participating batch; PP measurements cover each local stage.

  • Cache accounting: Expose supplemental LMCache reuse and record the decode consumer’s local prefix reuse after successful remote KV receipt. Reports display total reuse, GPU reuse, and LMCache reuse against a shared input-token denominator.

  • Metrics organization: Centralize GPU timing, scheduler instruments, histogram helpers, Prometheus initialization/HTTP export, snapshot collection/rendering, and request/stream instruments in atom/metrics/. Keep API composition, request middleware, streaming dispatch, and EngineStats with their owning components. The API injects its live stream-silence callback into the exporter, which reads it on the event loop before rendering in a worker thread. Preserve metric names, buckets, labels, multiprocess initialization order, and observation semantics; update imports, tests, and the metrics guide.

  • Streaming instrumentation: Share SSE parsing between request logging and streaming TTFT detection, while preserving first-generated-output semantics.

  • Reports and CI: Add workload, cache, queue, and GPU panels; role/category filters; independent prefill/decode instance selectors; and corresponding table/CSV data. Aggregate distributions from histogram buckets and ratios from raw counts. Increase engine refresh and
    CI scrape cadence to one second.

GPU timing is disabled by default and can be enabled with ATOM_ENABLE_METRICS_DEVICE_TIMER=1. Agentic CI enables it for both service roles, with environment overrides supported.
[glm-5.2-mxfp4-glm-52-mxfp4-1p1d-cpp4-dcp4-agentic-lmcache-1m-c48-nightly](https://github.com/ROCm/ATOM/actions/runs/34489056541/job/102928136600)
image
image

Test Plan

Focused tests cover:

  • Scheduler timing lifecycle, cache accounting, KV block counts, and PP scheduler deduplication.
  • GPU event polling/reuse, chunk accumulation, incomplete measurements, and worker snapshot routing.
  • Histogram consistency, collector registration, concurrent scrapes, and SSE first-output detection.
  • Report generation, PromQL aggregation, per-instance data, and CI collection configuration.

Test Result

  • Focused metrics, request timing, streaming, and report tests after the shared infrastructure migration: 288 passed, 2 skipped in freshly pulled rocm/atom-dev:latest. Skips require a GPU or an explicitly configured Prometheus binary. Prometheus bootstrap import-order and package-independence checks also passed.
  • CI-report and GC regression checks after restoring readiness scripts: 83 passed, 1 skipped (overlaps the focused suite). All four affected shell scripts pass bash -n.
  • Black and diff checks passed. Ruff introduced no findings; existing diagnostics in engine_core.py and api_server.py are unchanged.
  • Real inference with the migrated code in freshly pulled rocm/atom-dev:latest (2026-09-15 build, digest sha256:38714402a1b5a408d2a57d8f9d5900afe31b33302375b0981ee34509e36c3773): GLM-5.2 MXFP4, CPP4 prefill + TP4/DCP4 decode, Mooncake, MTP 3, prefix caching, LMCache CPU offload, and GPU timing enabled. The container's Prometheus client was upgraded from 0.23.1 to the PR-required 0.25.0.
  • 12/12 requests succeeded, covering streaming, non-streaming, an 8,466-token prompt split into three prefill chunks, repeated-prefix reuse, and eight concurrent requests. Input/output totals were 26,028/1,216 tokens. 36 metric checks and 13 migration checks passed with 33 live scrape rounds and eight concurrent scrapes per endpoint. Migration checks compare metric family names/types/help, sample names and label names against the prior live run, and verify GC and live stream-silence reporting.
  • Verified request/token accounting, all eight GPU workers, once-per-request prefill timing, 12 PD transfer observations, TTFT/ITL, histogram consistency, KV block partitions, idle stability, MTP counters, and zero exporter refresh errors. Warm-prefix reuse matched the response's 8,464 cached tokens; LMCache saved 8,448 tokens.
  • Mesh logged failed HuggingFace tokenizer lookups for the served-model alias; all backend requests and metric checks succeeded. The image also retains optional dependency upper-bound conflicts after installing the PR-required Prometheus client.
  • CPU-cache reload was not exercised because warm requests hit HBM. No full model-accuracy or comparative performance benchmark was run. Independent PD services currently emit engine_role="default"; the CI report distinguishes prefill/decode using the scrape target's role label.

GPU telemetry is asynchronous. The live test verified settled idle observations; ensuring final observations are included before every report collection stops remains a follow-up.

Copilot AI lite review requested due to automatic review settings September 10, 2026 06:51
@github-actions

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

  • ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

  • ✅ Run after the PR is approved and Pre Checkin passes
  • ✅ Run immediately when an approval review is submitted
  • ✅ Can be requested before approval with labels
Label Tests
ci:full Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom Run native ATOM model accuracy tests
ci:vllm Run ATOM vLLM OOT model accuracy tests
ci:sglang Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2186 --add-label <label>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core runtime scheduling/metrics behavior, API SSE timing/logging, and the CI report/query layer simultaneously, requiring careful end-to-end validation in real deployments.

Pull request overview

This PR expands ATOM’s observability pipeline end-to-end (engine → API /metrics → Prometheus → offline HTML report) by adding scheduler-, cache-, workload-, and GPU-forward-focused metrics, while refactoring the OpenAI server metrics exporter into a “compose metrics from owners” model to keep definitions close to their producing components.

Changes:

  • Add scheduler-owned cumulative histograms and gauges for queue time, decode batch size, PD KV wait, KV block states, and workload token histograms; propagate per-rank snapshots through the existing metrics snapshot path.
  • Add non-blocking GPU device-event forward timing with bounded buffering, plus per-request aggregated prefill GPU forward time (chunk-aware) exported per worker.
  • Update the metrics exporter and report generator to support explicit component registration, multi-instance breakdowns, cache tier coverage semantics, and improved SSE parsing/TTFT handling.
File summaries
File Description
tests/test_scheduler.py Updates scheduler cache-hit-rate wording to match new semantics.
tests/test_scheduler_metrics.py Adds lifecycle tests covering scheduler queue/KV wait timing, histograms, and snapshot/export invariants.
tests/test_observability_report.py Extends report query and validation tests (units, gauges, grouped queries, timestamps, cache/block counts).
tests/test_histogram.py Adds concurrency and correctness tests for weighted histogram exposition.
tests/test_gpu_metrics.py Adds tests for nonblocking GPU timing, request aggregation across chunks, and worker snapshot routing.
tests/test_gc_utils.py Adjusts GC scrape-cost wording to reference the new collector class.
tests/test_ci_latency_reports.py Expands CI report fixtures to cover new panels, instance breakdown, cache tiers, and KV block counts.
tests/entrypoints/test_streaming_dispatch.py Switches ITL tests to the new metrics composition API and shared bucket definitions.
tests/entrypoints/test_stream_silence_watchdog.py Updates logging tests to exercise the real client-stream boundary behavior.
tests/entrypoints/test_request_timing.py Reworks TTFT tests around middleware eligibility + stream-level parsing; adds more SSE shapes.
tests/entrypoints/test_metrics.py Refactors exporter tests for snapshot pinning, component registration, registry isolation, and concurrent scrapes.
docs/agentic_metrics.md Documents new metrics definitions, boundaries, tiers, and report semantics (incl. multi-instance).
atom/utils/histogram.py Introduces cumulative and weighted histogram helpers shared by metrics owners.
atom/utils/gc_utils.py Adds GCMetricsCollector for O(1) live GC scrape metrics.
atom/model_engine/sequence.py Adds per-sequence state used to track initial prefill GPU chunk timing boundaries.
atom/model_engine/scheduler.py Integrates scheduler metrics (enqueue, KV wait lifecycle, forward recording) and freezes prefill chunk boundaries.
atom/model_engine/scheduler_metrics.py Adds scheduler cumulative metric collection + Prometheus exposition adapter.
atom/model_engine/pp_engine_core.py Pushes periodic metrics from PP procs; records scheduler forward observations at PP head only.
atom/model_engine/model_runner.py Adds GPUForwardMetrics installation, worker snapshot collection, and forward timing decoration.
atom/model_engine/llm_engine.py Aggregates per-rank scheduler/worker metrics and preserves cache tier semantics for missing vs zero.
atom/model_engine/gpu_metrics.py Implements bounded, nonblocking device-event timing + per-request prefill aggregation + Prometheus exposition adapter.
atom/model_engine/engine_utility.py Pushes worker forward snapshots and augments scheduler snapshot with queue/KV-state semantics.
atom/model_engine/engine_stats.py Adds Prometheus exposition adapter for EngineStats + cache tier optionality handling.
atom/model_engine/engine_core.py Stamps request receipt time earlier, records scheduler forward observations, and increases metrics push cadence.
atom/model_engine/dp_metrics.py Adds Prometheus exposition adapter for DP router statistics.
atom/model_engine/async_proc.py Routes worker forward-metrics snapshots out-of-band without polluting KV/forward result queues.
atom/kv_transfer/offload/metrics.py Adds Prometheus exposition adapter for offload connector cumulative statistics.
atom/entrypoints/openai/streaming_dispatch.py Adds StreamMetrics (ITL + silence gauge) with shared ITL bucket constants.
atom/entrypoints/openai/sse.py Adds SSE frame inspection helper for parsing coalesced/complete local frames.
atom/entrypoints/openai/request_timing.py Moves generated-output detection into a reusable helper and adds RequestMetrics for TTFT instrumentation.
atom/entrypoints/openai/metrics.py Refactors exporter into a snapshot cache + explicit registry composition model with scrape snapshot pinning.
atom/entrypoints/openai/metrics_setup.py Adds a single composition point to register component collectors and local instruments per API instance.
atom/entrypoints/openai/api_server.py Switches to composed metrics, unifies stream parsing for logging+TTFT, and updates refresh cadence.
.github/scripts/atomesh/observability/README.md Updates report documentation to cover new panels, units, instance selectors, and cache tier semantics.
.github/scripts/atomesh/observability/export_report.py Expands panels, adds grouped-by-instance query support, cache/block count series, and stricter validation.
.github/scripts/atomesh/observability/collect_metrics.py Adjusts scrape cadence and report metadata/empty-report generation to match new panel/statistic model.
Review details
  • Files reviewed: 37/37 changed files
  • Comments generated: 0
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot AI review requested due to automatic review settings September 10, 2026 08:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It introduces cross-cutting observability changes spanning engine scheduling, worker telemetry routing, API streaming parsing, and CI report generation that warrant a final human validation pass for runtime/operational impact.

Review details
  • Files reviewed: 32/32 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 08:40

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It makes broad, coupled changes across the scheduler/engine loops, API streaming instrumentation, and Prometheus/reporting pipeline that warrant a human review for runtime overhead, compatibility, and operational rollout risk.

Review details
  • Files reviewed: 32/32 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 09:11

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It makes cross-cutting changes to scheduling/engine loops, OpenAI streaming parsing, and introduces new 1s-cadence metrics plus optional GPU event timing that warrant manual validation and performance/regression review.

Review details
  • Files reviewed: 36/36 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 09:28

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It changes core engine scheduling loops, streaming response parsing, and the metrics/reporting pipeline broadly, so it needs a final human review with real workload verification (perf + correctness) before approval.

Review details
  • Files reviewed: 36/36 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 10:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

It touches core scheduling/engine loops, streaming response handling, and CI observability/reporting in a performance-sensitive path, so it warrants final human review despite strong test coverage.

Review details
  • Files reviewed: 38/38 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 10:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

A confirmed resource-leak bug exists in GPUForwardMetrics.measure() where event pairs are not returned to the free pool on exceptions, risking unbounded CUDA/HIP event allocation under repeated failures.

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

atom/model_engine/gpu_metrics.py:115

  • In the exception path inside GPUForwardMetrics.measure(), the allocated event pair is neither appended to pending nor returned to free. If a forward raises repeatedly, this will keep allocating new CUDA/HIP events (until GC), increasing memory/handle pressure unnecessarily even though no samples are published for failures.
  • Files reviewed: 41/41 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 10, 2026 13:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

WeightedHistogram._child_samples() does not keep its lock held while iterating samples, which can allow scrapes to interleave with observations and read inconsistent histogram state.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Review details
  • Files reviewed: 41/41 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/utils/histogram.py Resolved
Comment thread atom/model_engine/engine_utility.py Resolved
@zufayu
zufayu requested a review from yitingw1 September 11, 2026 01:54
…ters

Record per-request and per-forward prefill work, decode context totals, and nonblocking GPU forward timings. Report GPU and LMCache reuse together, include pure-PD consumer cache accounting, and add independent service-instance filters.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Five moderate findings remain unresolved across DP readiness, remote metrics export, process initialization, metric labeling, and cache accounting.

Review details

Suppressed comments (5)

.github/scripts/atomesh/pd_server_atom.sh:518

  • For a multi-node DP service, api_server.main takes the dp_rank > 0 branch and exposes only /metrics on remote workers, so this probe can never succeed against those nodes. The coordinator paths below still pass every prefill_ips/decode_ips entry to this helper, which can make startup time out before the router and benchmark run; keep all worker addresses for Prometheus scraping, but probe only the coordinator API for each DP group (or skip remote DP entries).
  until python3 "${ATOMESH_SCRIPT_DIR}/../../../scripts/check_server_ready.py" "${url}" >/dev/null; do

atom/entrypoints/metrics.py:23

  • The worker-node metrics server registers only MultiProcessCollector, but the queue-state, KV-block, cache, and legacy engine metrics are materialized by _AtomMetricsCollector from engine snapshots, not written to multiprocess files. Consequently the remote DP endpoint used by the new per-instance report has no atom:scheduler_requests, atom:scheduler_kv_cache_blocks, or cache/state series, leaving those panels incomplete for remote instances. Export these state metrics natively or provide/update an equivalent snapshot collector on each worker.
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return start_http_server(port, addr=host, registry=registry)

atom/entrypoints/openai/api_server.py:387

  • openai_server.py is also re-executed as the multiprocessing spawn main module, so every EngineCore child reaches this top-level API initialization even though it never serves HTTP. In multiprocess mode this creates RequestMetrics/StreamMetrics mmap files (including zero-valued API histogram children) in every engine process, and the API's MultiProcessCollector scans them on each scrape. Keep API-only instruments out of spawned engine processes by guarding the API import/initialization to the serving process; otherwise scrape overhead and native metric storage scale with all engine workers.
_metrics_exporter, _request_metrics, _stream_metrics = create_metrics_exporter()

atom/model_engine/scheduler.py:520

  • The first SchedulerMetrics argument is named and exported as dp_rank, but this expression folds pipeline_parallel_rank into it. For a DP=0, PP=1 worker the scheduler series therefore says dp_rank="1", while the GPU series for the same worker says dp_rank="0", pp_rank="1"; grouping or correlating metrics by dp_rank misattributes the stage. Preserve the actual DP rank and add a separate pp_rank label (or rename this label to an explicitly flattened worker rank).
        self.waiting: deque[Sequence] = deque()
        self.running: deque[Sequence] = deque()
        self.config = config
        pc = getattr(config, "parallel_config", None)
        self.metrics = SchedulerMetrics(
            getattr(pc, "data_parallel_rank", 0)

atom/model_engine/scheduler.py:2207

  • _schedule_prefill_seq is called for every chunk/re-admission (including the chunked-prefill paths above), so this increments EngineStats cache totals once per chunk rather than once per request. A long prompt will therefore overcount requests, full-token denominators, and reuse; guard this admission with a request-level marker (and preserve the marker across preemption) or record it at the one-time cache-admission transition instead.
        num_batched_tokens: int,
  • Files reviewed: 50/50 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 16, 2026 05:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved findings affect coordinator-only readiness probes, per-node snapshot metrics, KV block accounting, GPU shutdown draining, and service-role labeling.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (6)

.github/scripts/atomesh/pd_server_atom.sh:1363

  • In the ordinary multi-node layout, prefill_ips contains one address per prefill node, while only the coordinator serves /v1/models; the other DP processes take the metrics-only branch. This loop therefore waits for a model list that remote workers never expose and can time out before the router starts. Wait only on the prefill coordinator; retain all node addresses as metrics targets.
    wait_api_ready "http://${prefill_ips[$idx]}:${prefill_ports[$idx]}/v1/models" \
      "prefill-${prefill_ips[$idx]}:${prefill_ports[$idx]}" \
      "${WAIT_SERVER_TIMEOUT}" "${server_pid}"

.github/scripts/atomesh/pd_server_atom.sh:1368

  • The decode address list in the ordinary multi-node layout also includes every DP node, but non-coordinator decode workers expose metrics only. This readiness loop consequently waits for /v1/models on endpoints that cannot answer it and blocks startup. Restrict the readiness probe to the decode coordinator while continuing to scrape every address for metrics.
    wait_api_ready "http://${decode_ips[$idx]}:${decode_ports[$idx]}/v1/models" \
      "decode-${decode_ips[$idx]}:${decode_ports[$idx]}" \
      "${WAIT_SERVER_TIMEOUT}"

atom/entrypoints/metrics.py:18

  • This endpoint is used for every non-coordinator DP target, but it exposes only the native multiprocess collector. The report's queue/block/cache families (atom:scheduler_requests, atom:scheduler_kv_cache_blocks, and atom:prefix_cache_*) are emitted by the snapshot collector in openai/metrics.py, not by the worker instruments. As a result, scraping the added remote targets omits their snapshot state and undercounts multi-node cache reuse and utilization. Publish those snapshot families on each node or keep them out of the per-node aggregation.
    """Expose this node's native instruments, without coordinator snapshots."""

atom/model_engine/block_pool.py:154

  • The pool's own invariant says these containers can retain stale IDs and that _free membership determines whether a block is actually free. Counting _cached and _protected directly therefore overcounts evictable blocks after claim/retirement churn (and can make the derived vacant count negative), so the exported KV partition becomes incorrect. Keep the free-set/hash filter here.
        return len(self._cached) + len(self._protected)

atom/model_engine/engine_utility.py:340

  • call_func("poll_forward_metrics") is fire-and-forget, so shutdown can terminate the runner after this request has been queued but before a final device-event poll observes events that complete after the last periodic tick. The final Prometheus scrape can consequently omit the last GPU samples (and incomplete request-prefill timings); add a shutdown drain/final poll that waits for the RPC to execute and gives pending events a bounded final observation window.
        # Poll ready device events even after the final forward. Workers write
        # native metrics directly; this RPC has no response payload.
        if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
            self.runner_mgr.call_func("poll_forward_metrics")

atom/model_engine/model_runner.py:833

  • engine_role is set to "default" unless config.enable_rapidserve is true. The PD launcher starts its services with --kv-transfer-config but does not enable --enable-rapidserve by default, so an ordinary agentic PD run records GPU histograms under role="default"; the report queries role="prefill" and role="decode" and therefore shows empty GPU panels despite device timing being enabled. Author must derive/propagate the actual prefill/decode service role independently of the RapidServe feature flag.
                engine_role=(
                    ("decode" if config.disagg_is_decode else "prefill")
                    if config.enable_rapidserve
                    else "default"
                ),
  • Files reviewed: 50/50 changed files
  • Comments generated: 1
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 16, 2026 06:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical metric-series and rank-labeling defects remain, alongside moderate reporting, readiness, metrics-lifecycle, and shutdown issues.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (8)

.github/scripts/atomesh/observability/collect_metrics.py:262

  • finalize_report only inspects the aggregate records/series fields and ignores p["instances"]. If an aggregate query fails while the grouped per-instance queries succeed, this adds a false “No samples” error (and can mark the run partial/unavailable) even though the report has valid data. Include instance bundles in the sample check.
        if not p.get("records")
        and not any(
            any(v is not None for _, v in points) for points in p["series"].values()
        )

.github/scripts/atomesh/observability/export_report.py:149

  • [verified] This always adds a prefill_kv_blocks panel, but the real disaggregated prefill path uses PrefillScheduler, whose block_manager is explicitly None; EngineUtilityHandler.collect_metrics() therefore emits no kv_blocks_* state for that role. The PromQL query will return no samples and this panel is permanently empty for every normal P/D deployment. Author must either omit the prefill block panel or provide a real supported source for prefill block state.
                    **common,
                    "id": f"{role}_kv_blocks",
                    "title": f"{role.title()} KV block utilization",
                    "detail": "Used, evictable cached, and vacant blocks · capacity-weighted across pools",
                    "metric": "atom:scheduler_kv_cache_blocks",
                    "unit": "%",
                    "kind": "blocks",

.github/scripts/atomesh/pd_server_atom.sh:1363

  • The prefill_ips/decode_ips arrays are also used as all-node metrics targets, and in a multi-node DP deployment can include non-coordinator ranks. Those ranks take the new dp_rank > 0 path and serve only /metrics, so probing /v1/models here can never succeed and rank-0 startup times out. Readiness should probe only each service coordinator; keep every address for metrics collection.
    wait_api_ready "http://${prefill_ips[$idx]}:${prefill_ports[$idx]}/v1/models" \
      "prefill-${prefill_ips[$idx]}:${prefill_ports[$idx]}" \
      "${WAIT_SERVER_TIMEOUT}" "${server_pid}"

atom/entrypoints/metrics.py:23

  • This endpoint is used by the dp_rank > 0 startup path, but it registers only MultiProcessCollector. The existing queue, KV/cache, and offload metrics are emitted by AtomMetricsExporter from LLMEngine snapshots, and that remote path never starts the snapshot refresh/lifespan. In a multi-node DP run, scraping these advertised remote targets therefore omits those panels and undercounts cache reuse; expose the snapshot exporter/refresh on each node or aggregate those snapshots into the coordinator before serving.
    registry = CollectorRegistry()
    multiprocess.MultiProcessCollector(registry)
    return start_http_server(port, addr=host, registry=registry)

atom/entrypoints/openai/api_server.py:2960

  • create_engine() constructs and returns an LLMEngine after its cores become ready; it does not block for the worker's lifetime. Therefore this finally runs immediately, shutting down the only metrics HTTP server, while the returned engine is discarded (and its child processes are not owned by a live reference). DP ranks greater than zero will stop exposing metrics as soon as startup completes. Keep the engine/server alive until the worker terminates, and close both during shutdown.
            engine_args.create_engine(tokenizer=tokenizer)
        finally:
            server.shutdown()
            server.server_close()

atom/metrics/scheduler.py:118

  • The decode counterpart has the same unbounded per-request cardinality: every request_id/sequence_id/started_at tuple remains in native multiprocess storage for the entire service lifetime. This can make the default /metrics endpoint and its backing files grow without bound; use bounded/opt-in request telemetry or an expiry path here as well.
        self.decode_request_context_tokens = Gauge(
            "atom:decode_request_context_tokens",
            "Logical context length at first real decode dispatch, once per request sequence; retained until metrics storage is cleaned.",
            [*labels, "request_id", "sequence_id", "started_at"],
            multiprocess_mode="max",
            registry=registry,

atom/model_engine/engine_core.py:346

  • If the shutdown request is consumed immediately after the last forward, this loop exits before another scheduled push_metrics call. Device events from that final forward may therefore never be polled or exported before the worker is destroyed, so the report's final scrape can silently miss the tail of GPU timing data. Add a shutdown-time poll/drain or otherwise keep collection alive until pending observations settle.
                now = time.monotonic()
                if now >= next_metrics_push:
                    next_metrics_push = now + metrics_interval
                    self.utility_handler.push_metrics()

atom/model_engine/scheduler.py:3471

  • This duplicate constructor applies the same PP offset to the dp_rank label, so disaggregated prefill scheduler metrics are mislabelled under PP. Keep this label consistent with the data-parallel rank used by the engine-core manager; pipeline stage identity is already a separate label for GPU metrics.
        self.metrics = SchedulerMetrics(
            getattr(pc, "data_parallel_rank", 0)
            * getattr(config, "pipeline_parallel_size", 1)
            + getattr(pc, "pipeline_parallel_rank", 0),
            self._METRICS_ROLE or "default",
  • Files reviewed: 51/51 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/metrics/scheduler.py Resolved
Comment thread atom/model_engine/scheduler.py Resolved
Copilot AI review requested due to automatic review settings September 16, 2026 07:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Three moderate findings remain unresolved, along with one documentation nit.

Review details

Suppressed comments (4)

atom/model_engine/engine_utility.py:340

  • If shutdown is consumed before the next periodic push_metrics() tick, the worker exits without another poll_forward_metrics call. CUDA events from the last forward can therefore remain pending and the final scrape/report silently omits the last GPU step (and possibly the completed request-prefill sample), even though the service has stopped cleanly. Add a bounded worker-side drain/poll during engine shutdown before the metrics directory is collected.
        # Poll ready device events even after the final forward. Workers write
        # native metrics directly; this RPC has no response payload.
        if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
            self.runner_mgr.call_func("poll_forward_metrics")

atom/model_engine/scheduler.py:3470

  • [verified] The same composite-rank label is used by PrefillScheduler; its snapshots are also keyed by data-parallel rank, so PP+DP prefill scheduler series get the wrong dp_rank label for every nonzero DP rank. Author must keep this label equal to data_parallel_rank and leave pipeline rank as a separate dimension if it is needed.
            getattr(pc, "data_parallel_rank", 0)
            * getattr(config, "pipeline_parallel_size", 1)
            + getattr(pc, "pipeline_parallel_rank", 0),

atom/model_engine/scheduler.py:522

  • [verified] This labels the scheduler with a composite dp_rank (dp * pp_size + pp_rank), but the metrics snapshot is keyed and aggregated by the actual data-parallel rank, and PP downstream stages do not publish scheduler snapshots. In a PP+DP deployment, for example DP rank 1 with pp_size=2 is exported as dp_rank="2", so the label no longer identifies the DP rank and can misattribute/skip per-rank scheduler series. Author must use the actual data-parallel rank here (PP is already represented by its own label where applicable).
            getattr(pc, "data_parallel_rank", 0)
            * getattr(config, "pipeline_parallel_size", 1)
            + getattr(pc, "pipeline_parallel_rank", 0),

docs/metrics_guide.md:271

  • This rule contradicts the request-context gauges documented above and implemented in atom/metrics/scheduler.py, which intentionally use request_id labels so completed requests can be reported. Qualify the rule to exempt those explicitly documented per-request gauges; otherwise the metric-addition guidance tells future contributors to remove a required label.
  • Files reviewed: 46/46 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

Copilot AI review requested due to automatic review settings September 16, 2026 07:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Two critical and five moderate findings remain unresolved, including duplicate Prometheus registrations, missing metric exports, and incomplete GPU-event draining.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

atom/entrypoints/openai/metrics_setup.py:26

  • prometheus_client treats an explicit registry=None as “do not register this instrument”; it does not mean “use the default native registry.” In a multiprocess launch this leaves the request/stream histograms out of both the default registry and the mmap files, so MultiProcessCollector cannot export TTFT or ITL at all. Register these instruments in the native default REGISTRY (while keeping exporter.registry for the snapshot collector) instead of passing None.
        multiprocess.MultiProcessCollector(registry)
        # Register only the multiprocess collector, not its instruments too.
        registry = None
    request_metrics = RequestMetrics(registry)

atom/metrics/gpu.py:38

  • This has the same registration problem for GPU telemetry: ModelRunner constructs GPUForwardMetrics without a registry, and the explicit registry=None suppresses registration of both histograms. The worker can observe and locally collect these objects, but no native multiprocess file is written, so the GPU panels remain empty in the real service. Register the default instance in the native registry, with an injectable isolated registry for tests.
class GPUForwardMetrics:
    def __init__(
        self,
        event_factory,
        max_pending=256,
        max_requests=4096,
        *,
        dp_rank=0,
        pp_rank=0,
        tp_rank=0,
        engine_role="default",
        registry=None,

atom/metrics/prometheus.py:18

  • This endpoint registers only MultiProcessCollector, while the new atom:scheduler_requests and atom:scheduler_kv_cache_blocks families are emitted by the API's snapshot collector from EngineUtilityHandler.collect_metrics, not by native worker instruments. In the dp_rank > 0 startup path this server therefore cannot expose queue/block state (or the other snapshot-only cache/state metrics), so scraping every configured DP instance still produces incomplete per-instance panels. Publish the snapshot collector on remote nodes or explicitly exclude these metrics from those targets.
    """Expose this node's native instruments, without coordinator snapshots."""

atom/metrics/scheduler.py:55

  • The scheduler creates SchedulerMetrics without a registry, and this default is passed explicitly to every Histogram/Gauge. In prometheus_client, that disables registration, so the scheduler histograms and request-context gauges never reach the native multiprocess files or /metrics; the snapshot only contains queue/block state and cannot replace them. Use the process's native default registry (or explicitly inject one) for the production default, while allowing isolated registries in tests.
class SchedulerMetrics:
    def __init__(self, dp_rank=0, engine_role="default", *, registry=None):
        labels = {"dp_rank": str(dp_rank), "engine_role": engine_role}

atom/model_engine/engine_utility.py:340

  • [verified] This only enqueues poll_forward_metrics through the non-blocking RPC (wait_out=False); the engine shutdown paths then stop the workers without a final bounded poll/drain. Any forward whose end event is still incomplete at the last periodic poll is therefore never observed, so the final report can permanently omit its per-step and per-request GPU samples even after the final scrape. Author must add a bounded shutdown drain (or otherwise wait for pending device events) before terminating the workers and cover a forward that completes after the last periodic poll.
        # Poll ready device events even after the final forward. Workers write
        # native metrics directly; this RPC has no response payload.
        if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
            self.runner_mgr.call_func("poll_forward_metrics")
  • Files reviewed: 45/45 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread atom/metrics/gpu.py
pp_rank=0,
tp_rank=0,
engine_role="default",
registry=None,
self.running: deque[Sequence] = deque()
self.config = config
pc = getattr(config, "parallel_config", None)
self.metrics = SchedulerMetrics(
Copilot AI review requested due to automatic review settings September 16, 2026 07:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Five unresolved moderate findings block approval.

Review details

Suppressed comments (5)

.github/scripts/atomesh/observability/export_report.py:123

  • [verified] In the real disaggregated PD path, PrefillScheduler has no block manager and does not perform prefix-cache accounting, so it cannot emit the prefill KV-block/cache families. Creating these panels for both roles therefore produces empty prefill panels and causes otherwise successful PD reports to be marked partial; the fixture tests synthesize data that the prefill service does not produce. Author must omit decode-owned panels from the prefill role or source their values from the decode owner.
    roles = ("standalone",) if deployment == "standalone" else ("prefill", "decode")
    for role in roles:
        selector = f'job="atom",role="{role}"'
        common = {"selector": selector, "label": f"{role.upper()} · SCHEDULER"}
        panels.extend(

.github/scripts/atomesh/observability/export_report.py:366

  • [verified] The cache coverage guard compares GPU/LMCache series only with prompt series. If an entire target is missing, all of those series disappear for that instance, so the counts still match and the aggregate percentage is calculated from the surviving instances instead of remaining unknown, contrary to the documented incomplete-instance behavior. Author must include configured/up target coverage in this check or propagate missing coverage.
    if panel.get("cache_breakdown") and state in {"gpu", "lmcache"}:
        # A partial rollout/missing tier must not silently lower total reuse.
        # Zero counters from a service without LMCache remain valid samples.
        inputs = increase(metrics["prompt"])
        result = f"({result} and ({count}({values}) == {count}({inputs})))"

atom/model_engine/engine_utility.py:340

  • [verified] poll_forward_metrics is invoked only by periodic metric pushes; the shutdown/finalization paths do not issue a final poll or drain before the worker exits. A forward whose device event becomes ready after the last tick remains pending, so the final GPU report can omit its step and completed prefill-request sample. Author must add a bounded final poll/drain before worker shutdown.
        # Poll ready device events even after the final forward. Workers write
        # native metrics directly; this RPC has no response payload.
        if envs.ATOM_ENABLE_METRICS_DEVICE_TIMER and self.runner_mgr is not None:
            self.runner_mgr.call_func("poll_forward_metrics")

atom/model_engine/scheduler.py:523

  • [verified] The scheduler histogram label is computed as data_parallel_rank * pipeline_parallel_size + pipeline_parallel_rank, but metrics snapshots and LLMEngine.get_metrics_statistics() are keyed by the data-parallel rank only; downstream PP stages suppress scheduler snapshots. With DP=2 and PP=2, head observations are labeled 0 and 2 while the corresponding snapshot ranks are 0 and 1, so per-DP histograms are misidentified. Author must use the data-parallel rank here (and add a separate PP label only if stage-level scheduler observations are later emitted).
        self.metrics = SchedulerMetrics(
            getattr(pc, "data_parallel_rank", 0)
            * getattr(config, "pipeline_parallel_size", 1)
            + getattr(pc, "pipeline_parallel_rank", 0),
            self._METRICS_ROLE or "default",

atom/model_engine/scheduler.py:3471

  • [verified] This second constructor repeats the PP-folded value, so PrefillScheduler/other scheduler histograms use the same incorrect rank mapping described above. In a PP deployment the native metric labels no longer line up with the rank keys used by the aggregator, which can mis-group per-rank data. Author must pass the data-parallel rank directly here as well.
        self.metrics = SchedulerMetrics(
            getattr(pc, "data_parallel_rank", 0)
            * getattr(config, "pipeline_parallel_size", 1)
            + getattr(pc, "pipeline_parallel_rank", 0),
            self._METRICS_ROLE or "default",
  • Files reviewed: 45/45 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

@valarLip
valarLip merged commit b3cccda into main Sep 16, 2026
49 of 51 checks passed
@valarLip
valarLip deleted the Jasen/metrics2 branch September 16, 2026 11:17
xysheng-AMD added a commit to xysheng-AMD/ATOM that referenced this pull request Sep 17, 2026
…ded by

`Run non-GPU unit tests` has been red on main since ROCm#2186 landed. The two
changes are individually green and collide on merge: ROCm#2028 made
`Scheduler.preempt` strip the placeholder tokens postprocess appended, reading
`seq.num_placeholder_tokens`, and ROCm#2186 added a test that drives `preempt` with
a `SimpleNamespace` carrying two fields.

    strip = seq.num_placeholder_tokens
    E AttributeError: SimpleNamespace object has no attribute num_placeholder_tokens

The double now carries the three the strip width is bounded by, each at the
value a real `Sequence` starts it at, which leaves the width at zero and the
test asserting what it always asserted.

Unrelated to the KV budget change in this branch; it is here because it blocks
the branch. Reproduced on main at 709c25b alone -- the scheduled Pre Checkin
run 35156585585 failed the same job at the same step with no commit of ours in
it.

Co-authored-by: Cursor <cursoragent@cursor.com>
sajandhy pushed a commit to sajandhy/ATOM that referenced this pull request Sep 17, 2026
…m#2186)

* feat(metrics): add agentic scheduler metrics and report filters

* feat(metrics): add cache reuse and workload metrics with instance filters

Record per-request and per-forward prefill work, decode context totals, and nonblocking GPU forward timings. Report GPU and LMCache reuse together, include pure-PD consumer cache accounting, and add independent service-instance filters.

* feat(metrics): add per-request decode context distribution

* feat(metrics): sum prefill GPU forward time per request

* refactor(metrics): share client stream parsing for logging and TTFT

* refactor(metrics): define instruments in owning components

Move metric definitions and observation interfaces into request, stream,
scheduler, GPU, engine, DP, offload and GC modules. Compose them explicitly
through the native Prometheus registry and keep the exporter focused on
cached snapshots and refresh health.

Pin one snapshot per render and reserve optional family names during
registration. Extend the standard Histogram for token-weighted ITL while
preserving metric names, labels, buckets, cumulative values and snapshot IPC.
Document how to add and register component metrics.

Validation in rocm/atom-dev:nightly_202609071454:
- Broader regression: 2588 passed, 56 skipped, 3 xfailed.
- prometheus-client 0.25.0 focused suite: 180 passed, 1 GPU skip.
- Six-scenario exposition parity on 0.23.1 and 0.25.0; full snapshot
  preserves all 80 metric families and 1202 samples.

* refactor(metrics): limit additions to dashboard metrics

* docs: remove agentic metrics reference

* perf(metrics): make GPU timing opt-in and poll events in FIFO order

* feat(metrics): add prefill context length metrics and dashboard panels

* docs(metrics): document every exported metric and how to add one

Restores, in English, the metric reference and registration how-to that was
removed wholesale with docs/agentic_metrics.md in 5cd92d8 -- that delete
targeted the agentic framing and took the contributor-facing guide with it,
leaving the collector contract described only in a docstring.

Covers all 72 exported metric families grouped by owning component and the two
ways a number reaches /metrics: an in-process instrument registered against the
exporter registry, or a snapshot collector whose engine-side accumulator rides
the existing METRICS push and is converted to families at render time.

The how-to gives runnable examples for both paths plus the collector contract:
collect(None) declares every possible family without I/O, collect(snapshot)
only reads, and a missing snapshot field must not render as zero.

Indexed from the CLAUDE.md guide table, and the cross-link 5cd92d8 dropped
from the CI observability README is restored.

* fix(metrics): remove pool scans and correct histogram bounds

Maintain reusable free-block counts incrementally, extend batch context buckets, and reserve original collector names during registration. Cover pool lifecycle transitions, large contexts, metric collisions, and deterministic LMCache dependency failure.

* fix(metrics): render scrapes off the API event loop

* perf(metrics): simplify GPU histograms and recycle timing events

Pool prefill, decode and mixed forward steps into one per-worker GPU
histogram while retaining independent cumulative prefill-request timing.
Keep worker identities, histogram buckets and report percentiles; update
HTML titles and document the removal of the phase label.

Recycle timing event pairs after failed forwards, preserve exception
propagation and exclude failed samples from both distributions.

Validate pooled histogram parity, request accumulation, event reuse and
Prometheus-to-HTML integration: 98 passed, one real-device test skipped.

* fix(metrics): make update and scrape intervals configurable

Use one validated ATOM_METRICS_UPDATE_INTERVAL_S setting for ordinary,
DP and PP engine metrics pushes and API snapshot refresh, defaulting to
one second. Read the shared setting once when each loop starts.

Configure Prometheus scraping independently with --scrape-interval-seconds
and a shared function/CLI default. Bound scrape timeouts and adapt collector
waits and histogram query windows to longer intervals. Document both settings.

Validation: 160 focused tests passed, including all engine loop variants,
API lifespan, invalid values and real Prometheus collection at 500 ms.

* refactor(metrics): derive reusable block count from pool queues

* docs(metrics): clarify GPU timer initialization and graph capture

* refactor(metrics): use native multiprocess histograms

* refactor(metrics): inherit standard histogram observe and collection

* fix: validate API model lists before reporting readiness

Use a shared model-list probe for local and native ATOM CI startup gates so DP worker metrics responses cannot report readiness. Document coordinator probing for multi-node DP and cover invalid responses and startup transitions.

* fix: fall back to default for invalid metrics intervals

Warn and use the default one-second interval when ATOM_METRICS_UPDATE_INTERVAL_S is invalid, allowing startup to continue. Preserve valid overrides and cover fallback behavior in environment parsing, all engine loops, and API startup.

* fix: expose first decode context once per request via metrics

* fix: expose prefill context once per request via metrics

* fix: select meaningful gauge panel headlines

* fix: avoid retaining shared-cache prefill requests in metrics

* fix: time GPU forwards during hidden-state extraction

* fix: preserve newer request timing on stale event errors

* test: import current checkout in spawned metrics workers

* style: fix rollout model runner import grouping

* docs: explain metrics-only prefill GPU tracking

* refactor: move runtime metrics into atom.metrics

* refactor: centralize shared metrics infrastructure

* chore: drop readiness probe changes from metrics PR

* test: trim redundant metrics regression coverage

* test: consolidate TTFT scenarios and drop redundant query count
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants