Skip to content

[Feat] frontend: Support X-Priority and RFC 9218 Priority headers - #49604

Open
anencore94 wants to merge 5 commits into
sgl-project:mainfrom
anencore94:devin/1794236400-priority-sglang-headers-main-20261110
Open

anencore94 wants to merge 5 commits into
sgl-project:mainfrom
anencore94:devin/1794236400-priority-sglang-headers-main-20261110

Conversation

@anencore94

@anencore94 anencore94 commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Allow trusted gateways to provide request scheduling priority through X-Priority or RFC 9218 Priority without rewriting the JSON body. This frontend-only branch preserves SGLang's existing body/default handling, scheduling direction, and opt-in legacy x-override-priority behavior.

Modifications

  • Resolve priority on Python /v1/chat/completions and /v1/completions: explicit body priority, including 0 and null, precedes the new headers. A selected single integer X-Priority precedes explicit RFC urgency u=0..7, mapped to u * 10.
  • Add tri-state --allow-header-priority / --no-allow-header-priority. The new headers require --enable-priority-scheduling; explicit allow does not enable scheduling itself.
  • Use http-sf>=1.5.0,<2 for Structured Fields dictionaries, including repeated fields, duplicate keys, unknown members and parameters. Malformed selected X-Priority returns HTTP 400; malformed/invalid RFC urgency falls through to existing defaults. Existing signed 64-bit validation remains in effect.
  • Add registered CPU route-to-engine handoff coverage and rendered server-argument documentation, including the ingress trust boundary. No scheduler or metric schema change is included.

Higher integers still run first by default. Use --schedule-low-priority-values-first when lower RFC urgency should run first. The existing opt-in legacy override retains its chat-only scope and precedence. Responses, gateway forwarding and the Rust frontend are outside this feature.

Accuracy Tests

Parser dependency regression (latest change)

Validated on 2026-11-11 at commit 4be0ceb5f313199bde8f56d4ea45ecd9ae8060fd on Linux x86_64, Python 3.12.12, PyTorch 2.14.0+cpu and Transformers 5.19.0.

  • Raise the parser floor to http-sf>=1.5.0,<2 in all five serving manifests (default, CPU, NPU, Other runtime base and XPU); all five TOML files parse and have the same requirement.
  • Extend test_invalid_rfc_urgency_is_ignored with Priority: u=1, x=@. Both production OpenAI HTTP routes must return HTTP 200 and retain default priority 37, rather than returning HTTP 500. Generation is substituted by the existing fixture.
  • Negative control: with only http-sf downgraded to 1.0.4, the same test has 2 failing HTTP subtests, one per route, both reporting 500 != 200 / index out of range; the other 44 subtests pass.
  • Restore the exact new floor, http-sf==1.5.0: 12 tests and 122 subtests passed, with 196 dependency/runtime/deprecation warnings. All applicable changed-file pre-commit hooks passed, including at commit time; git diff --check passed.
PYTHONPATH=python /home/ubuntu/.venvs/sglang-priority-cpu/bin/python -m pytest \
  test/registered/unit/entrypoints/test_priority_headers.py -q --tb=short

PATH=/home/ubuntu/.venvs/sglang-lint312/bin:$PATH \
/home/ubuntu/.venvs/sglang-lint312/bin/python -m pre_commit run --files \
  python/pyproject.toml python/pyproject_cpu.toml python/pyproject_npu.toml \
  python/pyproject_other.toml python/pyproject_xpu.toml \
  test/registered/unit/entrypoints/test_priority_headers.py

This focused run is not a full-suite, device-specific installation, real-model GPU serving or current-head upstream CI validation. Earlier broader results below remain tied to their recorded commits.

Earlier frontend review fixes

Merged upstream main 1cfc8128d226c8f2b16d589313d791193b36b917 into the PR branch and reran the full suite below. The tested head for that run was efc09ec5fe4ee36bab9ccacab4521a517c1031ed.

  • Declared http-sf>=1.0.4,<2 in each serving manifest (default, CPU, NPU, Other runtime base and XPU). Checked the five declarations and imported the header module against a fresh isolated install of the parser. Full device-specific installations were outside this smoke test.
  • Resolve priority in chat/completion conversion, including Unlimited-OCR v2 split-stream windows. The real HTTP route regression reproduced 37 != -5 before the fix. The patched path delivers both header types to generation, preserves explicit body priority and returns HTTP 400 for a malformed selected header before SSE starts. The test supplies a deterministic window plan and substitutes generation.
  • 403 tests and 575 subtests passed in the Linux ARM64 CPU run. Applicable changed-file hooks passed.
PYTHONPATH=python SGLANG_USE_CPU_ENGINE=1 TIKTOKEN_ENCODINGS_BASE=/path/to/encodings \
python -m pytest \
  test/registered/unit/entrypoints/test_priority_headers.py \
  test/registered/cpu/test_request_headers.py \
  test/registered/unit/models/unlimited_ocr_v2/test_document_split.py \
  test/registered/unit/entrypoints/openai/test_serving_chat.py \
  test/registered/unit/entrypoints/openai/test_serving_completions.py \
  test/registered/unit/entrypoints/openai/test_serving_responses.py \
  test/registered/unit/entrypoints/openai/test_serving_responses_stream.py -q --tb=short

Earlier validation

The CPU, docs and Qwen3 GPU results below were recorded on c40ec119eb7f587eeb95fd1e7198fcfb52de2e5e, based on upstream daa171846f2dfc8162b4fbcfac13b073d7a6353a. Current-head GPU serving was outside this rerun.

Check Result
Header HTTP route/CLI/streaming/boundary suite 11 tests passed; included in the larger run below
Header, legacy routing headers, existing Chat/Completion/Responses serving suites 331 passed, 526 subtests passed
Negative control: restore only the common serving handoff to unmodified main, keep the new test Body/header precedence test fails in 10 subcases; patched path passes
Changed-file pre-commit All applicable hooks passed
mint validate Build passed, exit 0; 6 pre-existing warnings, identical on an unmodified-main docs snapshot
mint broken-links Passed: no broken internal links
Real-model GPU serving and streaming 48 HTTP cases passed across Chat Completions and Completions with header handling enabled and disabled
Real GPU scheduler queue order 2 checks passed, one per ordering direction

CPU regression environment: Linux ARM64, Python 3.12.16, PyTorch 2.14.1+cpu, Transformers 5.19.0, http-sf 1.5.0. The existing container was reused; it does not launch GPU kernels. Model generation is substituted in the new CPU test, while production routes, request parsing/rendering, conversion, serialization and default/range validation are real.

The expanded suites initially exposed missing and mismatched optional renderer dependencies. Matching tml-renderers==0.1.0 and openai-harmony==0.0.4 to python/pyproject.toml, then providing the official Harmony vocabulary cache, resolved those environment failures. The final full run above uses unchanged assertions and no new skips, retries or xfails.

Reproduction commands and documentation warnings

From the repository root in a Linux CPU test environment with the declared dependencies and official Harmony vocabularies available:

PYTHONPATH=python SGLANG_USE_CPU_ENGINE=1 \
TIKTOKEN_ENCODINGS_BASE=/path/to/official/encodings \
.venv/bin/python -m pytest \
  test/registered/unit/entrypoints/test_priority_headers.py \
  test/registered/cpu/test_request_headers.py \
  test/registered/unit/entrypoints/openai/test_serving_chat.py \
  test/registered/unit/entrypoints/openai/test_serving_completions.py \
  test/registered/unit/entrypoints/openai/test_serving_responses.py \
  test/registered/unit/entrypoints/openai/test_serving_responses_stream.py \
  -q --tb=short

cd docs
npx --yes mint validate
npx --yes mint broken-links

Documentation validation used Mintlify's full build validation. Both the branch and a separate unmodified-main documentation snapshot emit the same six warnings: one foreign admonition in Kimi-K2.5, two KaTeX warnings in speculative decoding, two foreign admonitions in cache_dit and one in models_with_ar. The modified server-arguments page introduces none. This is not a warning-free full-site build claim.

GPU validation used Qwen/Qwen3-0.6B at c1899de289a04d12100db370d81485cdf75e47ca on one NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb with CUDA 13.2, driver 595.91.07, Python 3.12.3 and PyTorch 2.14.1+cu132. The editable Python frontend build used SGLANG_BUILD_RUST_EXTS=none. The server used SGLANG_ENABLE_JIT_DEEPGEMM=0 because the container's NVCC 12.8 is below DeepGEMM's requirement. The virtual environment's ninja was on PATH for FlashInfer. Model generation ran on the GPU with deterministic inference and prefill/decode CUDA graphs disabled.

The localhost probe used --enable-priority-scheduling --disable-priority-preemption --max-running-requests 1. It compared successful non-streaming responses and usage to a plain request and verified streaming output, a finish reason and [DONE]. With automatic header handling, signed 64-bit endpoints succeeded and selected malformed or out-of-range X-Priority returned HTTP 400 on both routes. With --no-allow-header-priority, the same malformed headers were ignored.

For queue ordering, one streaming generation occupied the only running slot. /v1/loads?include=core confirmed two requests waiting with X-Priority: -5 and Priority: u=7. The requests completed in low-then-high order with --schedule-low-priority-values-first, and high-then-low order with the default direction and explicit --allow-header-priority. All 50 checks passed and the probe exited 0.

The initial upstream CI run was blocked by the draft gate (PR is draft. Blocking CI.). The PR is now open and non-draft. On current head 4be0ceb5f313199bde8f56d4ea45ecd9ae8060fd, GitHub lint passed, but Base CI failed at the approval gate with Missing required label 'run-ci'; the CPU/GPU test stages were skipped. Mintlify fork preview also requires approval from someone with write access. These are authorization gates, not passing runtime or remote documentation tests. CI state was checked again on 2026-11-12; upstream runtime tests and current-head GPU serving remain unverified.

Speed Tests and Profiling

No performance improvement is claimed. Scheduler ordering, preemption policy and model computation are unchanged. Parser and end-to-end serving overhead have not been benchmarked.

Related work was checked against the author's open PRs and priority/header keyword searches. #42740 forwards the existing legacy override through the model gateway; #41087 handles priority propagation on Responses. This branch adds two new ingress headers to chat/completions and does not replace either PR. No direct duplicate was identified in those searches.

At untrusted ingress, validate or strip client body priorities and priority headers before injecting authorized values. This feature does not enforce caller permissions or tenant quotas.

AI assistance (Devin/Codex) was used for implementation, review and validation. Metrics extraction and its compatibility/migration decisions remain separate from this PR.

Checklist

  • Format your code according to the contribution guide; all applicable changed-file pre-commit hooks passed.
  • Add unit tests with CPU CI registration.
  • Update rendered documentation; full build validation and internal link checks executed, with baseline warnings disclosed.
  • Record earlier Qwen3 real-model GPU validation.
  • Run current-head GPU serving validation.
  • Complete upstream CI.
  • Complete submitting-human code review.
  • Provide accuracy and speed benchmarks if claiming improvements; no such improvement is claimed here.
  • Follow the SGLang runtime-context and code style conventions in the changed frontend path.

Review and Merge Process

This PR is open and non-draft. Submitting-human review and upstream CI are pending. No additional reviewer requests, oncall pings or CI authorization commands were sent as part of this description update.


CI States

Latest PR Test (Base): ❌ Run #42942105462
Latest PR Test (Extra): ❌ Run #42942105262
Latest PR Test (AMD ROCm 10): ❌ Run #42942105378

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

Preserve explicit body priorities, legacy chat-only overrides, default priority handling and scheduling direction. Keep metrics and scheduler policy changes outside this branch. Apply to upstream daa1718.

Co-authored-by: Devin <devin-ai@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
@mintlify

mintlify Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Preview approval required

This pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit 4be0ceb.

Approve preview

@github-actions github-actions Bot added documentation Improvements or additions to documentation dependencies Pull requests that update a dependency file labels Nov 10, 2026
…94236400-priority-sglang-headers-main-20261110

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
Resolve header priorities during chat and completion request conversion so split-stream windows use the same priority precedence. Report malformed headers before sending SSE and declare the parser in all serving manifests.

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
@github-actions github-actions Bot added the npu label Nov 11, 2026
…94236400-priority-sglang-headers-main-20261110

Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies Pull requests that update a dependency file documentation Improvements or additions to documentation npu

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant