Skip to content

feat(model_loader): chunked fastsafetensors loading with an all_local option - #70

Draft
bobofang11235 wants to merge 2 commits into
apinge:qwen38-flash-next-amd-current-fix-installfrom
bobofang11235:coldstart/fastsafetensors-chunked
Draft

bobofang11235 wants to merge 2 commits into
apinge:qwen38-flash-next-amd-current-fix-installfrom
bobofang11235:coldstart/fastsafetensors-chunked

Conversation

@bobofang11235

@bobofang11235 bobofang11235 commented Oct 27, 2026 •

Copy link
Copy Markdown

Motivation

fastsafetensors_weights_iterator copies whole files to the device. A shard
larger than the memory left after model init fails with OOM: for
Qwen3.8-Flash-Next-PTPC-FP8-PLE-BF16 the first shard is 96 GB and only 78.48 GiB
is free per GPU at TP2 on MI308X (Tried to allocate 95.37 GiB).

Changes

  • SGLANG_FASTSAFETENSORS_MAX_BATCH_GIB: when set, load through fastsafetensors'
    ParallelLoader (requires fastsafetensors >= 0.5.0) with max_batch_bytes, so
    each file is read in chunks of that size.
  • Broadcasts run on a temporary NCCL group that is destroyed after loading. On the
    default WORLD group the communicator (0.51 GiB per GPU) stays allocated for the
    life of the process, because SGLang never uses that group otherwise, and the KV
    cache shrinks accordingly.
  • SGLANG_FASTSAFETENSORS_ALL_LOCAL=1: every rank reads all shards itself instead
    of receiving them by broadcast. On ROCm the broadcast path leaves about 0.47 GiB
    of per-hardware-queue scratch allocated per GPU; with all_local the KV cache
    matches the safetensors path.

Defaults are unchanged.

Accuracy

GSM8K on MI308X TP2, base branch c996fcb1e, 3 runs each with the safetensors loader (base) and with
this change (chunked fastsafetensors, all_local), 5-shot in one chat message, thinking off,
temperature 0, 1,314 scored questions:
python3 -m sglang.test.run_eval --eval-name gsm8k --api chat --num-examples 1319 --max-tokens 1024 --chat-template-kwargs '{"enable_thinking": false}'

run 1 run 2 run 3 mean
base (safetensors) 0.9673 0.9627 0.9665 0.9655
this PR (fastsafetensors, all_local) 0.9665 0.9673 0.9673 0.9670

The difference (+0.0015) is within run-to-run variation, as expected for bit-identical weights.

Results (MI308X, TP2, --model-loader-extra-config '{"enable_gds": false}')

Measured with the qwen4_exp load_weights fix applied, run back to back on the same GPUs:

loader weight load KV cache tokens
safetensors 77.6 s 3,318,272
fastsafetensors, chunked, broadcast 34.9 s 3,279,360
fastsafetensors, chunked, all_local 37.3 s 3,318,272

With FASTSAFETENSORS_ODIRECT=0 (reads through the page cache) and the qwen4_exp expert-mapping
index, all_local loading takes 25.5 s.

Peak device memory while reading the 96 GB shard: 15.6 GiB with 8 GiB chunks.

Validation

  • Compared every yielded tensor bit for bit against safetensors under TP2, in both
    broadcast and all_local mode: 149,318 tensors, 220.2 GiB, no differences.
  • Memory: the temporary group's communicator is fully released after loading;
    all_local leaves nothing behind.

Notes

  • GDS (hipFile) was slower than the host bounce-buffer path on this system
    (3.0 vs 11.6 GB/s), so the measurements use enable_gds: false.
  • all_local reads the whole checkpoint on every rank, so its I/O grows with TP size.
  • Requires fastsafetensors >= 0.5.0. 0.4.0 already accepts max_batch_bytes and
    all_local and yields bit-identical tensors, but this path is about 7x slower
    with it: the all_local iterator takes 48.7-49.5 s vs 6.3-6.6 s with 0.5.0 (same
    session), and server weight loading on the fork takes 74.1 s vs 26.0-27.1 s, about
    the same as safetensors (77.6 s). 0.3.x has no max_batch_bytes.

fastsafetensors_weights_iterator copies whole files to the device. A
checkpoint shard larger than the memory left after model init fails with
OOM: Qwen3.8-Flash-Next-PTPC-FP8-PLE-BF16 has a 96 GB shard and only
78 GiB is free per GPU on MI308X TP2.

With SGLANG_FASTSAFETENSORS_MAX_BATCH_GIB set, use fastsafetensors'
ParallelLoader (>= 0.5.0) with max_batch_bytes so each file is read in
chunks; peak device memory for the 96 GB shard is 15.6 GiB. Broadcasts
run on a temporary NCCL group that is destroyed after loading: on the
default group the communicator (0.5 GiB per GPU) would stay allocated for
the life of the process and shrink the KV cache.

Weight loading on MI308X TP2: 74 s (safetensors) -> 33-35 s. Every
yielded tensor was compared bit for bit against safetensors (149,318
tensors, 220 GiB, no differences). Default behaviour is unchanged.
…ading

SGLANG_FASTSAFETENSORS_ALL_LOCAL=1 makes every rank read all shards
itself instead of receiving them by broadcast. On ROCm the broadcast path
leaves about 0.47 GiB of per-hardware-queue scratch allocated per GPU;
without broadcasts the KV cache matches the safetensors path. At TP2 the
load time is about the same (37.3 s vs 34.9 s); the amount read grows
with the TP size. Every yielded tensor was compared bit for bit against
safetensors (no differences).
@yicyang

yicyang commented Oct 27, 2026

Copy link
Copy Markdown
Collaborator

Hi @bobofang11235 could you clarify the accuracy result after this PR, and add a performance test script as well for reproduce.

@apinge
apinge requested a review from yicyang October 29, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants