Skip to content

[Fix] glm53_flash: Give a step's first launch half the watchdog as its wait deadline (preview) - #10

Closed
heslami wants to merge 2 commits into
mainfrom
review/glm53-flash-entry-deadline
Closed

heslami wants to merge 2 commits into
mainfrom
review/glm53-flash-entry-deadline

Conversation

@heslami

@heslami heslami commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

Since #23, a routed decode step starts with layer 0's launch, which takes the step's tokens and sums the embedding rows over the ranks. So no RCCL collective, which has no deadline, meets the ranks before the step's first wait on a peer. A captured graph runs a step's launches back to back, so any host work between steps lands in that first launch. Its waits have the same deadline as every other wait: half of --watchdog-timeout, at most 60 s.

A cold SGLang JIT cache creates such host work. The first real decode builds clamp_position (ForwardBatch.init_new → load_jit) under an fcntl.flock in ~/.cache/sglang/jit. When that directory is on NFSv3 with local_lock=none, a rank blocked on the lock wakes only on its next 30 s poll. The ranks then reach their first routed step 0, 30 or 60 s apart. Past 60 s, the early ranks' first wait on a peer times out and the guard stops every GPU: a false fail-stop.

On main, 2 of 5 cold boots died this way during the server's own warmup request. The GPUs faulted at 0x0 in layer 0's launch (glm53_flash_layer_s2_tp4_k1_d0_q1_h1_n1_e1). A host stall of more than 60 s between steps, on any one rank, does the same.

Modifications

  • spec.py: the runtime that runs layer 0 gets max(ctx.deadline_s, 0.5 * watchdog_timeout) as its wait deadline. That runtime is the one with key.tokens, and layer 0 is always a step's first launch, since _build refuses a layer 0 that is not dense. Every other runtime keeps ctx.deadline_s.

    • The new deadline is half the watchdog, SGLang's own bound for a wait, without its 60 s cap. So a wait that never ends still stops every GPU before the watchdog fires.
    --watchdog-timeout every other launch layer 0's launch
    300 s (SGLang's default) 60 s 150 s
    600 s (the e2e test's) 60 s 300 s
    20 s (the e2e fault test's) 10 s 10 s
  • The spec reads get_device().watchdog_timeout directly and declares get_device and watchdog_timeout in SGLANG_READS. SGLang computes ctx.deadline_s from the same read: monokernels/runtime.py on ab580b2e06, kernel_plugins/amd/binding.py after [Refactor] kernels: Add the plugin kernel backend and split plugin contracts from their srt integration sgl-project/sglang#45389.

  • No kernel change. The deadline is a word of the launch's guard block, which the host writes at build (FlashRuntime.set_deadline).

  • test/unit/test_glm53_flash_spec.py: test_a_step_s_first_launch_waits_up_to_half_the_watchdog checks both deadlines at watchdogs of 300, 600 and 20 s. It takes the framework's deadline from SGLang's own deadline_s.

  • Second commit, test only, in the same file, promised in [Perf] [MI355X] glm53: Above 4 rows run each expert the step uses once in up/gate and down (preview, tables TBD) #27's review: test_the_spec_needs_aiters_grouped_topk re-implemented SGLang's env comparison, and test_sglang_refuses_its_jit_grouped_topk replaces it.

What this does not change:

  • A real stall in layer 0's launch now stops after half the watchdog (150 s at the default) instead of 60 s.
  • With --disable-cuda-graph, each launch is enqueued by its host, so host skew can reach any launch, and those launches keep 60 s.
  • The stall itself remains: this fix only removes the false stop. A SGLang JIT cache on local disk (SGLANG_JIT_CACHE_DIR) avoids the 30–60 s wait.

Accuracy Tests

No numerics change: there is no kernel or forward change.

Setup: MI355X, rocm/sgl-dev:v0.5.21-rocm10-mi35x-20261014 (sha256:e4498f5d595f, #22's and #23's image), SGLang main ab580b2e06, amd/GLM-5.3-Flash-Quark-MXFP4@b5688f25. Main is at 9484c60 and this PR at dd1e2c6. The boots run at TP4 with the e2e test's server flags. Each boot is a fresh server: its own warmup request makes the first routed step, then one 32-token greedy request follows.

Cold SGLang JIT cache (watchdog 600 s, cache removed before each boot, HOME on NFSv3 with local_lock=none; main and this PR alternate, 5 boots each). "Entry skew" is the last rank's entry into its first routed decode replay, measured from the first rank's. It comes from a per-rank host timeline that is out of tree.

Passed Entry skew (s)
main 3 of 5 failed: 60.5, and one with a rank still waiting on the lock when the GPUs stopped; passed: 30.5, 30.1, 30.2
this PR 5 of 5 60.7, 60.4, 60.6, 30.2, 30.3
  • On main, the GPUs stop 60.2 s and 60.0 s after the first rank enters, and the server never comes up.
  • With this PR, the early ranks' first step waits up to 60.7 s on the GPU, and the request completes.

Host delay, warm cache. Rank 3's host sleeps before its first routed decode replay, using the out-of-tree hook below. Main's 400 s case ran once, with the hook. Every other case ran twice: once with the timeline's delay and once with the hook.

Case Watchdog main this PR
90 s delay 600 s GPUs 0 to 2 fault at 0x0 in layer 0's launch 60.0 to 60.1 s after the others enter passes: the other ranks' first step takes 90.0 s on the GPU, and the request completes
400 s delay: a peer that never arrives 300 s stops at 60.1 s GPUs 0 to 2 fault at 0x0 in layer 0's launch 150.1 s after the others enter. The server exits before SGLang's watchdog fires

A stall past the first launch uses the repo's fault injection: watchdog 300 s and SGLANG_AMD_MONO_FAULT_INJECT=1:22:20, with the e2e fault test's request (2 × 1,024 tokens).

  • Rank 1 stalls at guarded wait 22 once 20 steps have run. Layer 0's launch and the dense launches have at most 21 guarded waits, and the KDA and sparse-attention launches have 22 to 51, so only a later launch can stall.
  • All four GPUs fault at 0x0 in a KDA launch (glm53_flash_layer_s2_tp4_k1_d0_q1_h1) 60.8 s after the request. No tokens are returned, and the server exits.
  • So the later launches keep the 60 s deadline.

Other checks on this head:

  • E2E: test_glm53_flash_server.py at TP4, 5 of 5 pass. This includes the fault case, where both deadlines are 10 s.
  • TP2: the spec builds and serves a request at --tp-size 2.
  • Code objects: compile-only builds of every kind (FFN-only, KDA, sparse attention, dense, layer 0, tail), at every width and at TP 2, 4 and 8, preshuffled and row-major FP8 where they apply. All 72 of 72 are byte-identical to main's.
  • Unit tests, run with --continue-on-collection-errors. Since [Feat] minimax_m31: Run MiniMax-M3.1-Flash's decode FFN as one monokernel launch a layer sgl-project/sgl-amd-mono#56, two MiniMax test files fail at collection on a CPU-only host, on main as here.
    • CPU only, SGLang ab580b2e06: 258 passed (main 257). On both, 3 contract cases fail: deepseek_v41's two, as before, and minimax_m31's.
    • CPU only, SGLang main 39b54ae0ee: 259 passed (main 258). On both, the minimax and minimax_m31 contract cases fail.
    • With a GPU visible, SGLang ab580b2e06: 290 passed (main 289), with the same 3 contract failures.
    • Negative controls: if _deadline returns the framework's deadline for every key, the deadline test fails (60.0 against 150.0). Without the variable in requires.env, the grouped top-k test fails.
The host-delay hook (out of tree): a sitecustomize.py on PYTHONPATH, with DELAY_RANK=3 DELAY_S=90
import importlib.abc
import importlib.util
import os
import sys
import time

MODULE = "sglang.srt.model_executor.runner.decode_cuda_graph_runner"


class _Delay(importlib.abc.MetaPathFinder):
    def find_spec(self, name, path, target=None):
        if name != MODULE:
            return None
        sys.meta_path.remove(self)
        spec = importlib.util.find_spec(name)
        load = spec.loader.exec_module

        def exec_module(module):
            load(module)
            execute, done = module.DecodeCudaGraphRunner.execute, []

            def delayed(self, forward_batch, *args, **kwargs):
                import torch.distributed as dist

                if not done and forward_batch.batch_size <= 8:
                    done.append(True)
                    if dist.get_rank() == int(os.environ["DELAY_RANK"]):
                        print(f"DELAY {time.time():.1f} rank {dist.get_rank()} sleeps "
                              f"{os.environ['DELAY_S']} s", file=sys.stderr, flush=True)
                        time.sleep(float(os.environ["DELAY_S"]))
                return execute(self, forward_batch, *args, **kwargs)

            module.DecodeCudaGraphRunner.execute = delayed

        spec.loader.exec_module = exec_module
        return spec


if os.environ.get("DELAY_RANK"):
    sys.meta_path.insert(0, _Delay())

Speed Tests and Profiling

Not measured. The code objects are byte-identical to main's. At run time, the only difference is the value of one guard word in layer 0's runtime, and a wait reads that word only on its periodic check while it is still pending.

Checklist

  • pre-commit run --all-files passes.
  • Tests under test/ cover the change.
  • The SGLang commit or PR this was tested against is named above (SGLang main ab580b2e06; the CPU unit tests also on 39b54ae0ee).
  • Accuracy and speed results are attached when kernel or forward code changes (no kernel or forward change).

Hassan Eslami added 2 commits October 25, 2026 15:29
… deadline

The ranks meet in a step's first launch (layer 0, from the tokens) after their
hosts' work between steps, and no collective without a deadline meets them before
its first wait on a peer. A cold SGLang JIT cache builds clamp_position at the first
decode behind a file lock, which on an NFS home re-polls every 30 s, so the ranks can
enter that launch more than 60 s apart and the early ranks' waits stop every GPU.
The runtime of layer 0 now gets half the watchdog timeout as its deadline (the
framework's deadline without its 60 s cap); every other runtime keeps the
framework's deadline. No kernel change.
The unit test now calls SGLang's monokernel_disable_reasons on the spec and
checks that setting SGLANG_OPT_USE_JIT_KERNEL_GROUPED_TOPK adds exactly that
refusal, instead of re-implementing the env comparison.
@heslami

heslami commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator Author

Published upstream: sgl-project#61.

@heslami heslami closed this Oct 26, 2026
@heslami
heslami deleted the review/glm53-flash-entry-deadline branch October 26, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant