Skip to content

[Fix] glm53_flash: Give a step's first launch half the watchdog as its wait deadline (preview, round 2) - #13

Closed
heslami wants to merge 2 commits into
review/base-da5faf9from
review/glm53-flash-entry-deadline-2
Closed

heslami wants to merge 2 commits into
review/base-da5faf9from
review/glm53-flash-entry-deadline-2

Conversation

@heslami

@heslami heslami commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Motivation

Since #23, a routed decode step starts with layer 0's launch, which takes the step's tokens and sums the embedding rows over the ranks. So no RCCL collective, which has no deadline, meets the ranks before the step's first wait on a peer. A captured graph runs a step's launches back to back, so any host work between steps lands in that first launch. Its waits have the same deadline as every other wait: half of --watchdog-timeout, at most 60 s.

A cold SGLang JIT cache creates such host work. The first real decode builds clamp_position (ForwardBatch.init_new → load_jit) under an fcntl.flock in ~/.cache/sglang/jit. When that directory is on NFSv3 with local_lock=none, a rank blocked on the lock wakes only on its next 30 s poll. The ranks then reach their first routed step 0, 30 or 60 s apart. Past 60 s, the early ranks' first wait on a peer times out and the guard stops every GPU: a false fail-stop.

On main, 3 of 5 cold boots died this way during the server's own warmup request. The GPUs faulted at 0x0 in layer 0's launch (glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1). A host stall of more than 60 s between steps, on any one rank, does the same.

Modifications

  • spec.py: the runtime that runs layer 0 gets min(max(ctx.deadline_s, 0.5 * watchdog_timeout), guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ) as its wait deadline. That runtime is the one with key.tokens, and layer 0 is always a step's first launch, since _build refuses a layer 0 that is not dense. Every other runtime keeps ctx.deadline_s.

    • The new deadline is half the watchdog, SGLang's own bound for a wait, without its 60 s cap. So a wait that never ends still stops every GPU before the watchdog fires.
    • It is at most 5,497 s, the longest whole second that a wait's 32-bit deadline check holds (guard.MAX_DEADLINE_TICKS, 5,497.56 s). So from --watchdog-timeout 10994 on, and at inf, layer 0's deadline is 5,497 s, and guard.deadline_ticks takes it. Whole seconds keep the cap's conversion to ticks exact.
    --watchdog-timeout every other launch layer 0's launch
    300 s (SGLang's default) 60 s 150 s
    600 s (the e2e test's) 60 s 300 s
    20 s (the e2e fault test's) 10 s 10 s
    11,100 s 60 s 5,497 s
    inf 60 s 5,497 s
  • _build passes the cap to _deadline, so the deadline test runs on a CPU-only host: common/guard.py imports AITER, which needs a GPU to import.

  • The spec reads get_device().watchdog_timeout directly and declares get_device and watchdog_timeout in SGLANG_READS. SGLang computes ctx.deadline_s from the same read: monokernels/runtime.py on ab580b2e06, kernel_plugins/amd/binding.py after [Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration sgl-project/sglang#45389.

  • No kernel change. The deadline is a word of the launch's guard block, which the host writes at build (FlashRuntime.set_deadline).

  • test/unit/test_glm53_flash_spec.py: test_a_step_s_first_launch_waits_up_to_half_the_watchdog checks both deadlines at watchdogs of 300, 600, 20 and 11,100 s and inf. It takes the framework's deadline from SGLang's own deadline_s.

  • Second commit, test only, in the same file, promised in [Perf] [MI355X] glm53: Above 4 rows run each expert the step uses once in up/gate and down (preview, tables TBD) #27's review: test_the_spec_needs_aiters_grouped_topk re-implemented SGLang's env comparison, and test_sglang_refuses_its_jit_grouped_topk replaces it.

What this does not change:

  • A real stall in layer 0's launch now stops after half the watchdog (150 s at the default) instead of 60 s. With --watchdog-timeout inf, SGLang's watchdog never fires, and such a stall stops after 5,497 s.
  • With --disable-cuda-graph, each launch is enqueued by its host, so host skew can reach any launch, and those launches keep 60 s.
  • The stall itself remains: this fix only removes the false stop. A SGLang JIT cache on local disk (SGLANG_JIT_CACHE_DIR) avoids the 30–60 s wait.

Accuracy Tests

No numerics change: there is no kernel or forward change.

Setup: MI355X, rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014 (sha256:e4498f5d595f), SGLang main ab580b2e06, amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25. Main is at da5faf9 and this PR at ac9f232. The boots run at TP4 with the e2e test's server flags, each a fresh server: its own warmup request makes the first routed step, then one 32-token greedy request follows. Rank 3's host stalls before its first routed decode replay, using the out-of-tree hook in the first comment below.

Around the guard's limit (5 s stall on rank 3). Layer 0's deadline is 60 s on main at every watchdog here.

--watchdog-timeout Layer 0's deadline with this PR main this PR
10,900 s 5,450 s serves serves
11,100 s 5,497 s (the cap) serves serves
inf 5,497 s (the cap) serves serves

With inf, each scheduler's watchdog thread exits at start, on main as here (time.sleep(inf) raises OverflowError in SGLang's watchdog.py), so SGLang's watchdog never fires.

Host stall into the first routed step:

Case Watchdog Layer 0's deadline (main / this PR) main this PR
90 s stall 600 s 60 s / 300 s GPUs 0 to 2 fault at 0x0 in layer 0's launch 60.0 s after the stall starts; the server never comes up the first routed step waits out the stall; the server is up 90.1 s after it starts, and the request completes
400 s stall: a peer that never arrives 300 s 60 s / 150 s GPUs 0 to 2 fault at 0x0 in layer 0's launch at 60.1 s GPUs 0 to 2 fault at 0x0 in layer 0's launch at 150.0 s, before SGLang's watchdog

A stall past the first launch uses the repo's fault injection: watchdog 300 s and SGLANG_AMD_MONO_FAULT_INJECT=1:22:20, with the e2e fault test's request (2 × 1,024 tokens).

  • Rank 1 stalls at guarded wait 22 once 20 steps have run. Layer 0's launch and the dense launches have at most 21 guarded waits, and the KDA and sparse-attention launches have 22 to 51, so only a later launch can stall.
  • All four GPUs fault at 0x0 in a KDA launch (glm53_flash_layer_s2_tp4_k1_d0_q1_h1) 60.2 s after the request. No tokens are returned, and the server exits.
  • So the later launches keep the 60 s deadline.

Cold SGLang JIT cache (watchdog 600 s, the cache removed before each boot, HOME on NFSv3 with local_lock=none; main and this PR alternate, 5 boots each). "Entry skew" is the last rank's entry into its first routed decode replay, measured from the first rank's. It comes from a per-rank host timeline that is out of tree.

Passed Entry skew (s)
main 2 of 5 failed: three boots with a rank still waiting on the lock when the GPUs stopped; passed: 30.2, 58.7
this PR 5 of 5 58.2, 60.6, 60.6, 30.1, 60.3
  • On main, the GPUs stop 60.1, 60.0 and 60.0 s after the first rank enters, and the server never comes up.
  • With this PR, the early ranks' first step waits up to 60.6 s on the GPU, and the request completes.
  • One more boot of main is not counted: its server failed to open its device at start (hipErrorLaunchFailure in current_device()), before loading the model. Its rerun is the 30.2 s pass.

Other checks on this head:

  • E2E: test_glm53_flash_server.py at TP4 with HSA_DISABLE_COREDUMP_ON_EXCEPTION=1, 5 of 5 pass: GSM8K 0.975, mean KL against prefill 0.012 to 0.033 (bound 0.06), 21 of 21 returned hidden-state rows give the sampled token, and the fault case, where both deadlines are 10 s.
  • Code objects: compile-only builds of every kind (FFN-only, KDA, sparse attention, dense, layer 0, tail), at every width and at TP 2, 4 and 8, preshuffled and row-major FP8 where they apply: 72 a tree.
    • In one build of each tree, 70 of 72 are byte-identical to main's.
    • The other two vary from build to build on main itself. Over 7 builds of main and 6 of this PR, the TP4 sparse-attention builds at 2 and 8 rows (both layouts) each give the same 2 or 3 variants on both trees, and every other build never varies.
    • Every build of both trees has the same VGPRs, SGPRs, spills, LDS and scratch as main's.
  • Unit tests, run with --continue-on-collection-errors. Two MiniMax test files fail at collection on a CPU-only host, on main as here.
    • CPU only, SGLang ab580b2e06: 261 passed (main 260). On both, 3 contract cases fail: deepseek_v41's two and minimax_m31's.
    • CPU only, SGLang main 39b54ae0ee: 262 passed (main 261). On both, the minimax and minimax_m31 contract cases fail.
    • With a GPU visible, SGLang ab580b2e06: 293 passed (main 292), with the same 3 contract failures.
    • Negative controls, CPU only: without the min, the deadline test fails at 11,100 s (5,550.0 against 5,497), and with that case removed, at inf (inf against 5,497). If _deadline returns the framework's deadline for every key, it fails at 300 s (60.0 against 150.0). Without the variable in requires.env, the grouped top-k test fails.

Speed Tests and Profiling

Not measured. The code objects are byte-identical to main's. At run time, the only difference is the value of one guard word in layer 0's runtime, and a wait reads that word only on its periodic check while it is still pending.

Checklist

  • pre-commit run --all-files passes.
  • Tests under test/ cover the change.
  • The SGLang commit or PR this was tested against is named above (SGLang main ab580b2e06; the CPU unit tests also on 39b54ae0ee).
  • Accuracy and speed results are attached when kernel or forward code changes (no kernel or forward change).

Hassan Eslami added 2 commits October 27, 2026 10:22
… deadline

The ranks meet in a step's first launch (layer 0, from the tokens) after their
hosts' work between steps, and no collective without a deadline meets them before
its first wait on a peer. A cold SGLang JIT cache builds clamp_position at the first
decode behind a file lock, which on an NFS home re-polls every 30 s, so the ranks can
enter that launch more than 60 s apart and the early ranks' waits stop every GPU.
The runtime of layer 0 now gets half the watchdog timeout as its deadline (the
framework's deadline without its 60 s cap), at most the longest whole second the
guard's 32-bit check holds (5,497 s, from guard.MAX_DEADLINE_TICKS), so a watchdog
past 10,995 s or inf still builds. Every other runtime keeps the framework's
deadline. No kernel change.
The unit test now calls SGLang's monokernel_disable_reasons on the spec and
checks that setting SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK adds exactly that
refusal, instead of re-implementing the env comparison.
@heslami

heslami commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator Author

Thanks for rerunning everything, and for finding the overflow on main with #19. Both required items are fixed by one cap, and the branch is rebased onto main da5faf9 (#19 and sgl-project#64). The new head is ac9f232, still two commits. sgl-project#62 and sgl-project#63 have landed since; this PR still merges cleanly, and it changes only spec.py and its test.

Stacking with #19: does the deadline spill? No, not any more. With the cap, every deadline fits #19's 32-bit check, and sgl-project#64's deadline_ticks takes it:

  1. Bound. Layer 0's deadline is at most guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ = 5,497 s, and every other launch keeps SGLang's (at most 60 s). In the unit test, a watchdog of 11,100 s and inf both give 5,497 s. Without the cap, the test fails at 11,100 s (5550.0 == 5497), and with that case removed, at inf (inf == 5497).

  2. On GPU, main da5faf9 + this PR at TP4, each a fresh server with a 5 s stall on rank 3 before its first routed decode replay:

    --watchdog-timeout Layer 0's deadline Result
    600 s, with a 90 s stall instead 300 s waits out the stall: up 90.1 s after it starts, and the request completes
    10,900 s 5,450 s serves
    11,100 s 5,497 s (the cap) serves
    inf 5,497 s (the cap) serves

    On the same main, this PR's first version fails at build on every rank: at 11,100 s with ValueError: a 5550.0 s wait deadline does not fit the wait's 32-bit check; use at most 5497 s ([Fix] common: Refuse a wait deadline the 32-bit check cannot hold sgl-project/sgl-amd-mono#64), and at inf with OverflowError: cannot convert float infinity to integer. Main alone serves all of these too: its layer 0 deadline is 60 s.

  3. Kernels. This is a spec-only change: every kernel source file is identical to main's. In compile-only builds of all 72 keys per tree (every kind and width, TP 2/4/8, both FP8 layouts), every build has the same VGPRs, SGPRs, spills, LDS and scratch as main's. The bytes match too, apart from four TP4 sparse-attention builds that vary from build to build on main itself: over 7 builds of main and 6 of this PR, both trees give the same 2 or 3 variants.

Required

  1. Cap. Layer 0's deadline is now min(max(ctx.deadline_s, 0.5 * watchdog_timeout), guard.MAX_DEADLINE_TICKS // guard.CLOCK_HZ). The cap comes from guard, not a copied constant. Whole seconds keep its conversion to ticks exact.
    • _build passes the cap to _deadline, because common/guard.py imports AITER, which needs a GPU to import. So the deadline test still runs on a CPU-only host.
    • The test covers 300, 600, 20 and 11,100 s and inf.
  2. inf. The same cap gives 5,497 s, and the server comes up (table above). On main and here alike, each scheduler's watchdog thread exits at start (time.sleep(inf) raises OverflowError in SGLang's watchdog.py), which is why SGLang's watchdog never fires.

Nit

  • _deadline's docstring is one line: "A step's first launch waits up to half the watchdog, at most longest_s."

Re-validation on ac9f232 against main da5faf9 (your setup: SGLang ab580b2e06, TP4, a fresh server per run, your hook):

  • 90 s stall, watchdog 600 s: main: GPUs 0 to 2 fault at 0x0 in layer 0's launch 60.0 s after the stall starts. This PR: up 90.1 s after it starts, and the request completes.
  • 400 s stall, watchdog 300 s: this PR stops at 150.0 s (GPUs 0 to 2, layer 0's launch), before SGLang's watchdog; main at 60.1 s.
  • Later launches keep 60 s: with SGLANG_AMD_MONO_FAULT_INJECT=1:22:20 and watchdog 300 s, all four GPUs fault at 0x0 in a KDA launch 60.2 s after the request.
  • E2E: all 5 cases of test_glm53_flash_server.py pass at TP4 (GSM8K 0.975).
  • Cold JIT cache: main 2 of 5 boots pass, this PR 5 of 5, with entry skews up to 60.6 s.
  • Unit tests, CPU only: 261 passed (main 260), with the same three contract failures and the two MiniMax modules failing at collection, as on main. With a GPU visible: 293 passed (main 292).
  • Negative controls: besides the two above, if _deadline returns the framework's deadline for every key, the deadline test fails (60.0 == 150.0). Without the variable in requires.env, the grouped top-k test fails.

The description is updated for the new head: the cap, its table rows, and every number re-measured against da5faf9.

@heslami heslami closed this Oct 27, 2026
@heslami
heslami deleted the review/glm53-flash-entry-deadline-2 branch October 27, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant