Repository navigation
Conversation
… deadline The ranks meet in a step's first launch (layer 0, from the tokens) after their hosts' work between steps, and no collective without a deadline meets them before its first wait on a peer. A cold SGLang JIT cache builds clamp_position at the first decode behind a file lock, which on an NFS home re-polls every 30 s, so the ranks can enter that launch more than 60 s apart and the early ranks' waits stop every GPU. The runtime of layer 0 now gets half the watchdog timeout as its deadline (the framework's deadline without its 60 s cap), at most the longest whole second the guard's 32-bit check holds (5,497 s, from guard.MAX_DEADLINE_TICKS), so a watchdog past 10,995 s or inf still builds. Every other runtime keeps the framework's deadline. No kernel change.
The unit test now calls SGLang's monokernel_disable_reasons on the spec and checks that setting SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK adds exactly that refusal, instead of re-implementing the env comparison.
|
Thanks for rerunning everything, and for finding the overflow on main with #19. Both required items are fixed by one cap, and the branch is rebased onto main da5faf9 (#19 and sgl-project#64). The new head is ac9f232, still two commits. sgl-project#62 and sgl-project#63 have landed since; this PR still merges cleanly, and it changes only Stacking with #19: does the deadline spill? No, not any more. With the cap, every deadline fits #19's 32-bit check, and sgl-project#64's
Required
Nit
Re-validation on ac9f232 against main da5faf9 (your setup: SGLang ab580b2e06, TP4, a fresh server per run, your hook):
The description is updated for the new head: the cap, its table rows, and every number re-measured against da5faf9. |
Motivation
Since #23, a routed decode step starts with layer 0's launch, which takes the step's tokens and sums the embedding rows over the ranks. So no RCCL collective, which has no deadline, meets the ranks before the step's first wait on a peer. A captured graph runs a step's launches back to back, so any host work between steps lands in that first launch. Its waits have the same deadline as every other wait: half of
--watchdog-timeout, at most 60 s.A cold SGLang JIT cache creates such host work. The first real decode builds
clamp_position(ForwardBatch.init_new→load_jit) under anfcntl.flockin~/.cache/sglang/jit. When that directory is on NFSv3 withlocal_lock=none, a rank blocked on the lock wakes only on its next 30 s poll. The ranks then reach their first routed step 0, 30 or 60 s apart. Past 60 s, the early ranks' first wait on a peer times out and the guard stops every GPU: a false fail-stop.On main, 3 of 5 cold boots died this way during the server's own warmup request. The GPUs faulted at 0x0 in layer 0's launch (
glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1). A host stall of more than 60 s between steps, on any one rank, does the same.Modifications
spec.py: the runtime that runs layer 0 getsmin(max(ctx.deadline_s, 0.5 * watchdog_timeout), guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ)as its wait deadline. That runtime is the one withkey.tokens, and layer 0 is always a step's first launch, since_buildrefuses a layer 0 that is not dense. Every other runtime keepsctx.deadline_s.guard.MAX_DEADLINE_TICKS, 5,497.56 s). So from--watchdog-timeout 10994on, and atinf, layer 0's deadline is 5,497 s, andguard.deadline_tickstakes it. Whole seconds keep the cap's conversion to ticks exact.--watchdog-timeoutinf_buildpasses the cap to_deadline, so the deadline test runs on a CPU-only host:common/guard.pyimports AITER, which needs a GPU to import.The spec reads
get_device().watchdog_timeoutdirectly and declaresget_deviceandwatchdog_timeoutinSGLANG_READS. SGLang computesctx.deadline_sfrom the same read:monokernels/runtime.pyon ab580b2e06,kernel_plugins/amd/binding.pyafter [Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration sgl-project/sglang#45389.No kernel change. The deadline is a word of the launch's guard block, which the host writes at build (
FlashRuntime.set_deadline).test/unit/test_glm53_flash_spec.py:test_a_step_s_first_launch_waits_up_to_half_the_watchdogchecks both deadlines at watchdogs of 300, 600, 20 and 11,100 s andinf. It takes the framework's deadline from SGLang's owndeadline_s.Second commit, test only, in the same file, promised in [Perf] [MI355X] glm53: Above 4 rows run each expert the step uses once in up/gate and down (preview, tables TBD) #27's review:
test_the_spec_needs_aiters_grouped_topkre-implemented SGLang's env comparison, andtest_sglang_refuses_its_jit_grouped_topkreplaces it.monokernel_disable_reasonswith and withoutSGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK=1. Setting the variable must add exactly that refusal.What this does not change:
--watchdog-timeout inf, SGLang's watchdog never fires, and such a stall stops after 5,497 s.--disable-cuda-graph, each launch is enqueued by its host, so host skew can reach any launch, and those launches keep 60 s.SGLANG_JIT_CACHE_DIR) avoids the 30–60 s wait.Accuracy Tests
No numerics change: there is no kernel or forward change.
Setup: MI355X,
rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014(sha256:e4498f5d595f), SGLang main ab580b2e06,amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25. Main is at da5faf9 and this PR at ac9f232. The boots run at TP4 with the e2e test's server flags, each a fresh server: its own warmup request makes the first routed step, then one 32-token greedy request follows. Rank 3's host stalls before its first routed decode replay, using the out-of-tree hook in the first comment below.Around the guard's limit (5 s stall on rank 3). Layer 0's deadline is 60 s on main at every watchdog here.
--watchdog-timeoutinfWith
inf, each scheduler's watchdog thread exits at start, on main as here (time.sleep(inf)raisesOverflowErrorin SGLang'swatchdog.py), so SGLang's watchdog never fires.Host stall into the first routed step:
A stall past the first launch uses the repo's fault injection: watchdog 300 s and
SGLANG_AMD_MONO_FAULT_INJECT=1:22:20, with the e2e fault test's request (2 × 1,024 tokens).glm53_flash_layer_s2_tp4_k1_d0_q1_h1) 60.2 s after the request. No tokens are returned, and the server exits.Cold SGLang JIT cache (watchdog 600 s, the cache removed before each boot,
HOMEon NFSv3 withlocal_lock=none; main and this PR alternate, 5 boots each). "Entry skew" is the last rank's entry into its first routed decode replay, measured from the first rank's. It comes from a per-rank host timeline that is out of tree.hipErrorLaunchFailureincurrent_device()), before loading the model. Its rerun is the 30.2 s pass.Other checks on this head:
test_glm53_flash_server.pyat TP4 withHSA_DISABLE_COREDUMP_ON_EXCEPTION=1, 5 of 5 pass: GSM8K 0.975, mean KL against prefill 0.012 to 0.033 (bound 0.06), 21 of 21 returned hidden-state rows give the sampled token, and the fault case, where both deadlines are 10 s.--continue-on-collection-errors. Two MiniMax test files fail at collection on a CPU-only host, on main as here.min, the deadline test fails at 11,100 s (5,550.0 against 5,497), and with that case removed, atinf(inf against 5,497). If_deadlinereturns the framework's deadline for every key, it fails at 300 s (60.0 against 150.0). Without the variable inrequires.env, the grouped top-k test fails.Speed Tests and Profiling
Not measured. The code objects are byte-identical to main's. At run time, the only difference is the value of one guard word in layer 0's runtime, and a wait reads that word only on its periodic check while it is still pending.
Checklist
pre-commit run --all-filespasses.test/cover the change.