fix(benchmark): scrape every SGLang attention-DP process for custom AIPerf metrics

[weireweire]

Summary

Custom benchmarks (for example the InferenceX AgentX harness, which wraps AIPerf) receive AIPERF_SERVER_METRICS_URLS with logical worker leaders only. With SGLang attention data parallelism, that drops every follower node's DP ranks from the server-side metrics.

Root cause

_get_benchmark_env passes logical_workers_only=True for custom benchmarks, on the assumption that a multi-node worker's follower ranks duplicate the leader's engine. That is true for pure TP. With attention DP (dp-size > 1), each physical SGLang process schedules its own attention-DP ranks and exports distinct cache and load metrics for them. Scraping only the leader therefore covers part of the deployment: on a layout with 2-node DEP8 prefill workers and a 4-node DEP16 decode worker, prefill metrics covered DP0-3 of DP0-7 and decode covered 4 of 16 ranks, so server-side prefix-cache hit rates and load series were computed from a subset of ranks.

Fix

• New _custom_metrics_need_physical_processes(): true when the backend is SGLang, the worker is not a Dynamo sidecar, and any role sets dp-size > 1.
• In that case custom benchmarks get the physical-process metrics list, the same list built-in AIPerf runners already use.
• If no physical metrics ports can be enumerated (for example a single-node DEP worker), the logical leader URLs are kept.
• Unchanged: routing endpoints (SRT_*_ENDPOINTS, SRT_*_IPS) stay logical; pure-TP and sidecar workers keep logical metrics URLs; an explicit AIPERF_SERVER_METRICS_URLS in the recipe environment still wins.

Note for reviewers: this raises the number of endpoints AIPerf scrapes per cycle (one per worker node instead of one per worker). AIPerf's AIPERF_SERVER_METRICS_COLLECTION_INTERVAL (default 0.333 s) sets how often each endpoint is scraped; a worker leader's /metrics response can be several hundred KB, so deployments that care about scrape overhead may want a longer interval.

Validation

• New unit tests: attention-DP disaggregation gets leader and follower URLs for prefill and decode while routing endpoints stay logical; pure-TP and sidecar workers keep logical leaders.
• tests/test_benchmarks.py: all new tests pass; remaining failures also occur on main in the same environment.
• ruff check src/srtctl/ and ruff format --check pass.
• End to end on a GB300 DeepSeek-V4 disaggregated run (four 2-node DEP8 prefill workers, one 4-node DEP16 decode worker): AIPerf scrapes all 12 worker processes instead of 5 leaders, and the server-side metrics cover every attention-DP rank.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 88.88889% with 2 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@a154f37). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/cli/mixins/benchmark_stage.py	88.88%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #554   +/-   ##
=======================================
  Coverage        ?   83.96%           
=======================================
  Files           ?      153           
  Lines           ?    21773           
  Branches        ?        0           
=======================================
  Hits            ?    18282           
  Misses          ?     3491           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.