Parent: #1773 · Phase 1
Make GET /engine_metrics include gRPC workers so engine metrics consolidate across transports. Today the fetch is HTTP-only (worker/manager.rs:978-1012, fan_out :53-85); a gRPC worker yields grpc://host:port/metrics, fails, and is silently dropped (an all-gRPC fleet returns "All backend requests failed.").
Implementation
- Discover an HTTP metrics endpoint for gRPC workers during metadata discovery (
workflow/steps/local/discover_metadata.rs → create_worker.rs), from GetServerInfo.server_args (a full Struct, proto/sglang_scheduler.proto:461):
- Store as a label, not a
WorkerSpec field — WorkerSpec is in crates/protocols (sacred); only model_gateway uses this, so the iron law keeps it out of protocols. Add the keys to SGLANG_GRPC_KEYS (routers/grpc/client.rs:783); expose Worker::metrics_url() beside base_url() (worker/worker.rs:664).
- Branch the fetch on
connection_mode() in get_engine_metrics/fan_out (worker/manager.rs:53-85,978-1012), mirroring monitor.rs:627-632. HTTP keeps GET {url}/metrics; gRPC uses metrics_url(). Unknown-endpoint workers are skipped and counted.
- Aggregation unchanged (
worker/metrics_aggregator.rs).
New metric
smg_engine_metrics_scrape_total{connection_mode,result} (success/failure/skipped). Replace the blanket "All backend requests failed" when the real cause is "no gRPC endpoint known".
Acceptance criteria
- An all-gRPC fleet returns merged engine metrics from
/engine_metrics.
- Mixed HTTP+gRPC fleets merge both, labeled per worker.
- A gRPC worker with metrics disabled is skipped + counted, not an error.
Tests
- Unit:
server_args→metrics_url extraction (explicit / derived / disabled); fan_out URL per connection mode.
- Integration: gRPC stub advertising
metrics_url + stub /metrics → merged into /engine_metrics.
Note: W1 is always-on (a bug fix, no flag). Works for upstream engines that already expose /metrics without W4.
Parent: #1773 · Phase 1
Make
GET /engine_metricsinclude gRPC workers so engine metrics consolidate across transports. Today the fetch is HTTP-only (worker/manager.rs:978-1012,fan_out:53-85); a gRPC worker yieldsgrpc://host:port/metrics, fails, and is silently dropped (an all-gRPC fleet returns "All backend requests failed.").Implementation
workflow/steps/local/discover_metadata.rs→create_worker.rs), fromGetServerInfo.server_args(a full Struct,proto/sglang_scheduler.proto:461):metrics_url/prometheus_portkey (emitted by W4, [Metrics] W4: Python servicer HTTP /metrics sidecar #1777) → else derivehttp://{host}:{port}/metrics→ gated on anenable_metrics-style flag so a dark port is never scraped.WorkerSpecfield —WorkerSpecis incrates/protocols(sacred); onlymodel_gatewayuses this, so the iron law keeps it out of protocols. Add the keys toSGLANG_GRPC_KEYS(routers/grpc/client.rs:783); exposeWorker::metrics_url()besidebase_url()(worker/worker.rs:664).connection_mode()inget_engine_metrics/fan_out(worker/manager.rs:53-85,978-1012), mirroringmonitor.rs:627-632. HTTP keepsGET {url}/metrics; gRPC usesmetrics_url(). Unknown-endpoint workers are skipped and counted.worker/metrics_aggregator.rs).New metric
smg_engine_metrics_scrape_total{connection_mode,result}(success/failure/skipped). Replace the blanket "All backend requests failed" when the real cause is "no gRPC endpoint known".Acceptance criteria
/engine_metrics.Tests
server_args→metrics_urlextraction (explicit / derived / disabled);fan_outURL per connection mode.metrics_url+ stub/metrics→ merged into/engine_metrics.Note: W1 is always-on (a bug fix, no flag). Works for upstream engines that already expose
/metricswithout W4.