feat(gateway): add engine-agnostic observability metrics and overhaul Grafana dashboard

[varungup90]

Pull Request Description

Summary

• Add gateway-level gauges and counters for operational visibility: per-model in-flight requests (gateway_model_in_flight_requests), global in-flight (gateway_in_flight_requests), PD prefill outstanding (gateway_prefill_outstanding_requests), summable token usage (gateway_input_tokens_total, gateway_output_tokens_total, gateway_requests_with_usage_total), and ready engine replica counts (model_replicas).
• Improve latency metric accuracy for streaming: record first-token time on the initial response chunk, use it for TTFT buckets, emit TPOT buckets, and apply coarser end-to-end total-time bucketing.
• Overhaul the Grafana dashboard into an engine-agnostic SLO view with $model filtering, parameterized datasource, and collapsed developer internals; remove unused metrics (gateway_e2e_duration_seconds, aggregate success/fail counters, gateway_decode_outstanding_requests).

Details

Gateway Plugin

• Track and emit global and per-model in-flight request gauges for the lifetime of each request.
• Emit input/output token counters and a has_usage counter when usage is present in responses.
• Fix streaming TTFT by setting RoutingContext.FirstTokenTime on the first streamed body chunk and propagating Stream from the request body handler.
• Emit TPOT bucket metrics for PD and non-PD streaming paths.
• Use role-name pod label (fallback: ROLE_NAME env) for metric labels.

Cache

• Emit model_replicas=1 for each ready, routable engine pod (HTTP server on pod group index 0), with stale-series cleanup when pods leave the cache.

PD Disaggregation

• Track and emit gateway_prefill_outstanding_requests around prefill RPCs.

Prefix Cache

• Use _init_ instead of an empty model label for indexer status metrics.

Grafana

• Reorganize panels into Service Health, Latency SLOs, Token Throughput, Prefix Cache, PD Disaggregation, and collapsed Engine Internals / Gateway Infrastructure sections.
• Wire panels to the new and existing gateway metrics with $model template variable support.

Related Issues

Resolves: #[Insert issue number(s)]

Important: Before submitting, please complete the description above and review the checklist below.

Contribution Guidelines (Expand for Details)

We appreciate your contribution to aibrix! To ensure a smooth review process and maintain high code quality, please adhere to the following guidelines:

Pull Request Title Format

Your PR title should start with one of these prefixes to indicate the nature of the change:

• [Bug]: Corrections to existing functionality
• [CI]: Changes to build process or CI pipeline
• [Docs]: Updates or additions to documentation
• [API]: Modifications to aibrix's API or interface
• [CLI]: Changes or additions to the Command Line Interface
• [Misc]: For changes not covered above (use sparingly)

Note: For changes spanning multiple categories, use multiple prefixes in order of importance.

Submission Checklist

• PR title includes appropriate prefix(es)
• Changes are clearly explained in the PR description
• New and existing tests pass successfully
• Code adheres to project style and best practices
• Documentation updated to reflect changes (if applicable)
• Thorough testing completed, no regressions introduced

By submitting this PR, you confirm that you've read these guidelines and your changes align with the project's contribution standards.

[gemini-code-assist]

Code Review

This pull request introduces several enhancements to the gateway and model metrics tracking, including the addition of model replica metrics, in-flight request tracking per model, and token usage counters. It also improves the accuracy of streaming metrics like Time to First Token (TTFT) and Time Per Output Token (TPOT). The review feedback highlights two potential metric leak issues: one where a model name change during the request lifecycle could leak in-flight counts, and another where a panic during a prefill request could prevent the outstanding prefill counter from being decremented. Robust code suggestions using dynamic tracking and defer blocks are provided to address these concerns.

[Jeffwan]

lgtm

[Jeffwan]

@varungup90 there's some conflicts needs to be resolved