Repository navigation
[Feat] frontend: Support X-Priority and RFC 9218 Priority headers - #49604
Open
anencore94 wants to merge 5 commits into
Open
anencore94 wants to merge 5 commits into
anencore94 wants to merge 5 commits into
Conversation
Preserve explicit body priorities, legacy chat-only overrides, default priority handling and scheduling direction. Keep metrics and scheduler policy changes outside this branch. Apply to upstream daa1718. Co-authored-by: Devin <devin-ai@users.noreply.github.com> Co-authored-by: Codex <noreply@openai.com> Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
Contributor
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
…94236400-priority-sglang-headers-main-20261110 Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
Resolve header priorities during chat and completion request conversion so split-stream windows use the same priority precedence. Report malformed headers before sending SSE and declare the parser in all serving manifests. Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
…94236400-priority-sglang-headers-main-20261110 Signed-off-by: Jaeyeon Kim <anencore94@gmail.com>
anencore94
marked this pull request as ready for review
November 11, 2026 00:53
anencore94
requested review from
CatherineSue,
Fridge003,
JustinTong0323,
ispobock,
merrymercy,
slin1237,
sogalin,
wisclmy0611 and
zijiexia
as code owners
November 11, 2026 00:53
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This was referenced Nov 11, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Allow trusted gateways to provide request scheduling priority through
X-Priorityor RFC 9218Prioritywithout rewriting the JSON body. This frontend-only branch preserves SGLang's existing body/default handling, scheduling direction, and opt-in legacyx-override-prioritybehavior.Modifications
/v1/chat/completionsand/v1/completions: explicit body priority, including0andnull, precedes the new headers. A selected single integerX-Priorityprecedes explicit RFC urgencyu=0..7, mapped tou * 10.--allow-header-priority/--no-allow-header-priority. The new headers require--enable-priority-scheduling; explicit allow does not enable scheduling itself.http-sf>=1.5.0,<2for Structured Fields dictionaries, including repeated fields, duplicate keys, unknown members and parameters. Malformed selected X-Priority returns HTTP 400; malformed/invalid RFC urgency falls through to existing defaults. Existing signed 64-bit validation remains in effect.Higher integers still run first by default. Use
--schedule-low-priority-values-firstwhen lower RFC urgency should run first. The existing opt-in legacy override retains its chat-only scope and precedence. Responses, gateway forwarding and the Rust frontend are outside this feature.Accuracy Tests
Parser dependency regression (latest change)
Validated on 2026-11-11 at commit
4be0ceb5f313199bde8f56d4ea45ecd9ae8060fdon Linux x86_64, Python 3.12.12, PyTorch 2.14.0+cpu and Transformers 5.19.0.http-sf>=1.5.0,<2in all five serving manifests (default, CPU, NPU, Other runtime base and XPU); all five TOML files parse and have the same requirement.test_invalid_rfc_urgency_is_ignoredwithPriority: u=1, x=@. Both production OpenAI HTTP routes must return HTTP 200 and retain default priority37, rather than returning HTTP 500. Generation is substituted by the existing fixture.http-sfdowngraded to 1.0.4, the same test has 2 failing HTTP subtests, one per route, both reporting500 != 200/index out of range; the other 44 subtests pass.http-sf==1.5.0: 12 tests and 122 subtests passed, with 196 dependency/runtime/deprecation warnings. All applicable changed-file pre-commit hooks passed, including at commit time;git diff --checkpassed.PYTHONPATH=python /home/ubuntu/.venvs/sglang-priority-cpu/bin/python -m pytest \ test/registered/unit/entrypoints/test_priority_headers.py -q --tb=short PATH=/home/ubuntu/.venvs/sglang-lint312/bin:$PATH \ /home/ubuntu/.venvs/sglang-lint312/bin/python -m pre_commit run --files \ python/pyproject.toml python/pyproject_cpu.toml python/pyproject_npu.toml \ python/pyproject_other.toml python/pyproject_xpu.toml \ test/registered/unit/entrypoints/test_priority_headers.pyThis focused run is not a full-suite, device-specific installation, real-model GPU serving or current-head upstream CI validation. Earlier broader results below remain tied to their recorded commits.
Earlier frontend review fixes
Merged upstream main
1cfc8128d226c8f2b16d589313d791193b36b917into the PR branch and reran the full suite below. The tested head for that run wasefc09ec5fe4ee36bab9ccacab4521a517c1031ed.http-sf>=1.0.4,<2in each serving manifest (default, CPU, NPU, Other runtime base and XPU). Checked the five declarations and imported the header module against a fresh isolated install of the parser. Full device-specific installations were outside this smoke test.37 != -5before the fix. The patched path delivers both header types to generation, preserves explicit body priority and returns HTTP 400 for a malformed selected header before SSE starts. The test supplies a deterministic window plan and substitutes generation.Earlier validation
The CPU, docs and Qwen3 GPU results below were recorded on
c40ec119eb7f587eeb95fd1e7198fcfb52de2e5e, based on upstreamdaa171846f2dfc8162b4fbcfac13b073d7a6353a. Current-head GPU serving was outside this rerun.mint validatemint broken-linksCPU regression environment: Linux ARM64, Python 3.12.16, PyTorch 2.14.1+cpu, Transformers 5.19.0,
http-sf1.5.0. The existing container was reused; it does not launch GPU kernels. Model generation is substituted in the new CPU test, while production routes, request parsing/rendering, conversion, serialization and default/range validation are real.The expanded suites initially exposed missing and mismatched optional renderer dependencies. Matching
tml-renderers==0.1.0andopenai-harmony==0.0.4topython/pyproject.toml, then providing the official Harmony vocabulary cache, resolved those environment failures. The final full run above uses unchanged assertions and no new skips, retries or xfails.Reproduction commands and documentation warnings
From the repository root in a Linux CPU test environment with the declared dependencies and official Harmony vocabularies available:
PYTHONPATH=python SGLANG_USE_CPU_ENGINE=1 \ TIKTOKEN_ENCODINGS_BASE=/path/to/official/encodings \ .venv/bin/python -m pytest \ test/registered/unit/entrypoints/test_priority_headers.py \ test/registered/cpu/test_request_headers.py \ test/registered/unit/entrypoints/openai/test_serving_chat.py \ test/registered/unit/entrypoints/openai/test_serving_completions.py \ test/registered/unit/entrypoints/openai/test_serving_responses.py \ test/registered/unit/entrypoints/openai/test_serving_responses_stream.py \ -q --tb=short cd docs npx --yes mint validate npx --yes mint broken-linksDocumentation validation used Mintlify's full build validation. Both the branch and a separate unmodified-main documentation snapshot emit the same six warnings: one foreign admonition in Kimi-K2.5, two KaTeX warnings in speculative decoding, two foreign admonitions in cache_dit and one in models_with_ar. The modified server-arguments page introduces none. This is not a warning-free full-site build claim.
GPU validation used Qwen/Qwen3-0.6B at
c1899de289a04d12100db370d81485cdf75e47caon one NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb with CUDA 13.2, driver 595.91.07, Python 3.12.3 and PyTorch 2.14.1+cu132. The editable Python frontend build usedSGLANG_BUILD_RUST_EXTS=none. The server usedSGLANG_ENABLE_JIT_DEEPGEMM=0because the container's NVCC 12.8 is below DeepGEMM's requirement. The virtual environment'sninjawas on PATH for FlashInfer. Model generation ran on the GPU with deterministic inference and prefill/decode CUDA graphs disabled.The localhost probe used
--enable-priority-scheduling --disable-priority-preemption --max-running-requests 1. It compared successful non-streaming responses and usage to a plain request and verified streaming output, a finish reason and[DONE]. With automatic header handling, signed 64-bit endpoints succeeded and selected malformed or out-of-range X-Priority returned HTTP 400 on both routes. With--no-allow-header-priority, the same malformed headers were ignored.For queue ordering, one streaming generation occupied the only running slot.
/v1/loads?include=coreconfirmed two requests waiting withX-Priority: -5andPriority: u=7. The requests completed in low-then-high order with--schedule-low-priority-values-first, and high-then-low order with the default direction and explicit--allow-header-priority. All 50 checks passed and the probe exited 0.The initial upstream CI run was blocked by the draft gate (
PR is draft. Blocking CI.). The PR is now open and non-draft. On current head4be0ceb5f313199bde8f56d4ea45ecd9ae8060fd, GitHub lint passed, but Base CI failed at the approval gate withMissing required label 'run-ci'; the CPU/GPU test stages were skipped. Mintlify fork preview also requires approval from someone with write access. These are authorization gates, not passing runtime or remote documentation tests. CI state was checked again on 2026-11-12; upstream runtime tests and current-head GPU serving remain unverified.Speed Tests and Profiling
No performance improvement is claimed. Scheduler ordering, preemption policy and model computation are unchanged. Parser and end-to-end serving overhead have not been benchmarked.
Related work was checked against the author's open PRs and priority/header keyword searches. #42740 forwards the existing legacy override through the model gateway; #41087 handles priority propagation on Responses. This branch adds two new ingress headers to chat/completions and does not replace either PR. No direct duplicate was identified in those searches.
At untrusted ingress, validate or strip client body priorities and priority headers before injecting authorized values. This feature does not enforce caller permissions or tenant quotas.
AI assistance (Devin/Codex) was used for implementation, review and validation. Metrics extraction and its compatibility/migration decisions remain separate from this PR.
Checklist
Review and Merge Process
This PR is open and non-draft. Submitting-human review and upstream CI are pending. No additional reviewer requests, oncall pings or CI authorization commands were sent as part of this description update.
CI States
Latest PR Test (Base): ❌ Run #42942105462
Latest PR Test (Extra): ❌ Run #42942105262
Latest PR Test (AMD ROCm 10): ❌ Run #42942105378
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: