Repository navigation
Conversation
added 2 commits
October 25, 2026 15:29
… deadline The ranks meet in a step's first launch (layer 0, from the tokens) after their hosts' work between steps, and no collective without a deadline meets them before its first wait on a peer. A cold SGLang JIT cache builds clamp_position at the first decode behind a file lock, which on an NFS home re-polls every 30 s, so the ranks can enter that launch more than 60 s apart and the early ranks' waits stop every GPU. The runtime of layer 0 now gets half the watchdog timeout as its deadline (the framework's deadline without its 60 s cap); every other runtime keeps the framework's deadline. No kernel change.
The unit test now calls SGLang's monokernel_disable_reasons on the spec and checks that setting SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK adds exactly that refusal, instead of re-implementing the env comparison.
Collaborator
Author
|
Published upstream: sgl-project#61. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Since #23, a routed decode step starts with layer 0's launch, which takes the step's tokens and sums the embedding rows over the ranks. So no RCCL collective, which has no deadline, meets the ranks before the step's first wait on a peer. A captured graph runs a step's launches back to back, so any host work between steps lands in that first launch. Its waits have the same deadline as every other wait: half of
--watchdog-timeout, at most 60 s.A cold SGLang JIT cache creates such host work. The first real decode builds
clamp_position(ForwardBatch.init_new→load_jit) under anfcntl.flockin~/.cache/sglang/jit. When that directory is on NFSv3 withlocal_lock=none, a rank blocked on the lock wakes only on its next 30 s poll. The ranks then reach their first routed step 0, 30 or 60 s apart. Past 60 s, the early ranks' first wait on a peer times out and the guard stops every GPU: a false fail-stop.On main, 2 of 5 cold boots died this way during the server's own warmup request. The GPUs faulted at 0x0 in layer 0's launch (
glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1). A host stall of more than 60 s between steps, on any one rank, does the same.Modifications
spec.py: the runtime that runs layer 0 getsmax(ctx.deadline_s, 0.5 * watchdog_timeout)as its wait deadline. That runtime is the one withkey.tokens, and layer 0 is always a step's first launch, since_buildrefuses a layer 0 that is not dense. Every other runtime keepsctx.deadline_s.--watchdog-timeoutThe spec reads
get_device().watchdog_timeoutdirectly and declaresget_deviceandwatchdog_timeoutinSGLANG_READS. SGLang computesctx.deadline_sfrom the same read:monokernels/runtime.pyon ab580b2e06,kernel_plugins/amd/binding.pyafter [Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration sgl-project/sglang#45389.No kernel change. The deadline is a word of the launch's guard block, which the host writes at build (
FlashRuntime.set_deadline).test/unit/test_glm53_flash_spec.py:test_a_step_s_first_launch_waits_up_to_half_the_watchdogchecks both deadlines at watchdogs of 300, 600 and 20 s. It takes the framework's deadline from SGLang's owndeadline_s.Second commit, test only, in the same file, promised in [Perf] [MI355X] glm53: Above 4 rows run each expert the step uses once in up/gate and down (preview, tables TBD) #27's review:
test_the_spec_needs_aiters_grouped_topkre-implemented SGLang's env comparison, andtest_sglang_refuses_its_jit_grouped_topkreplaces it.monokernel_disable_reasonswith and withoutSGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK=1. Setting the variable must add exactly that refusal.What this does not change:
--disable-cuda-graph, each launch is enqueued by its host, so host skew can reach any launch, and those launches keep 60 s.SGLANG_JIT_CACHE_DIR) avoids the 30–60 s wait.Accuracy Tests
No numerics change: there is no kernel or forward change.
Setup: MI355X,
rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014(sha256:e4498f5d595f, #22's and #23's image), SGLang main ab580b2e06,amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25. Main is at 9484c60 and this PR at dd1e2c6. The boots run at TP4 with the e2e test's server flags. Each boot is a fresh server: its own warmup request makes the first routed step, then one 32-token greedy request follows.Cold SGLang JIT cache (watchdog 600 s, cache removed before each boot,
HOMEon NFSv3 withlocal_lock=none; main and this PR alternate, 5 boots each). "Entry skew" is the last rank's entry into its first routed decode replay, measured from the first rank's. It comes from a per-rank host timeline that is out of tree.Host delay, warm cache. Rank 3's host sleeps before its first routed decode replay, using the out-of-tree hook below. Main's 400 s case ran once, with the hook. Every other case ran twice: once with the timeline's delay and once with the hook.
A stall past the first launch uses the repo's fault injection: watchdog 300 s and
SGLANG_AMD_MONO_FAULT_INJECT=1:22:20, with the e2e fault test's request (2 × 1,024 tokens).glm53_flash_layer_s2_tp4_k1_d0_q1_h1) 60.8 s after the request. No tokens are returned, and the server exits.Other checks on this head:
test_glm53_flash_server.pyat TP4, 5 of 5 pass. This includes the fault case, where both deadlines are 10 s.--tp-size 2.--continue-on-collection-errors. Since [Feat] minimax_m31: Run MiniMax-M3.1-Flash's decode FFN as one monokernel launch a layer sgl-project/sgl-amd-mono#56, two MiniMax test files fail at collection on a CPU-only host, on main as here._deadlinereturns the framework's deadline for every key, the deadline test fails (60.0 against 150.0). Without the variable inrequires.env, the grouped top-k test fails.The host-delay hook (out of tree): a
sitecustomize.pyonPYTHONPATH, withDELAY_RANK=3 DELAY_S=90Speed Tests and Profiling
Not measured. The code objects are byte-identical to main's. At run time, the only difference is the value of one guard word in layer 0's runtime, and a wait reads that word only on its periodic check while it is still pending.
Checklist
pre-commit run --all-filespasses.test/cover the change.