Skip to content

[Metrics] W4: Python servicer HTTP /metrics sidecar #1777

Description

@slin1237

Parent: #1773 · Phase 3

Add an HTTP /metrics Prometheus sidecar to SMG's gRPC servicers so they're scrape-able in gRPC mode like upstream SGLang — "enable that for other engines" on the producer side. Today no servicer exposes /metrics (each serve_grpc starts only a grpc.aio.server); the SGLang /metrics referenced in motivation comes from upstream SGLang's HTTP sidecar (the on_request_manager_ready hook, grpc_servicer/.../sglang/server.py:37), not SMG's servicer.

Implementation

  • Shared sidecar in grpc_servicer: a small aiohttp/uvicorn /metrics endpoint on a configurable --metrics-port, started alongside each serve_grpc (sglang/, vllm/, tokenspeed/ server.py); bind/serve failure logged, must not abort the gRPC server. (MLX excluded — no PD; add later only if general engine metrics are wanted.)
  • Source: reuse each engine's native Prometheus registry where it exists (vLLM; SGLang via its sidecar hook), plus a shared collector re-exposing the scheduler load snapshot already computed for GetLoads — guarantees coverage for tokenspeed without bespoke work.
  • Close the W1 loop: advertise metrics_url/prometheus_port in the server_args struct each servicer reports (grpc_servicer/.../sglang/servicer.py:535-568) so W1 auto-discovers it.

Acceptance criteria

  • Each servicer serves valid Prometheus exposition on --metrics-port.
  • GetServerInfo.server_args advertises the metrics address.
  • The gRPC server is unaffected if the sidecar fails to bind.

Tests

  • pytest: /metrics returns valid exposition; server_args advertises the address.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or requestgrpcgRPC client and router changesmetrics-consolidationEpic: Engine Metrics Consolidation & PD Observability

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions