Repository navigation
feat(model_loader): chunked fastsafetensors loading with an all_local option - #70
Draft
bobofang11235 wants to merge 2 commits into
Conversation
fastsafetensors_weights_iterator copies whole files to the device. A checkpoint shard larger than the memory left after model init fails with OOM: Qwen3.8-Flash-Next-PTPC-FP8-PLE-BF16 has a 96 GB shard and only 78 GiB is free per GPU on MI308X TP2. With SGLANG_FASTSAFETENSORS_MAX_BATCH_GIB set, use fastsafetensors' ParallelLoader (>= 0.5.0) with max_batch_bytes so each file is read in chunks; peak device memory for the 96 GB shard is 15.6 GiB. Broadcasts run on a temporary NCCL group that is destroyed after loading: on the default group the communicator (0.5 GiB per GPU) would stay allocated for the life of the process and shrink the KV cache. Weight loading on MI308X TP2: 74 s (safetensors) -> 33-35 s. Every yielded tensor was compared bit for bit against safetensors (149,318 tensors, 220 GiB, no differences). Default behaviour is unchanged.
…ading SGLANG_FASTSAFETENSORS_ALL_LOCAL=1 makes every rank read all shards itself instead of receiving them by broadcast. On ROCm the broadcast path leaves about 0.47 GiB of per-hardware-queue scratch allocated per GPU; without broadcasts the KV cache matches the safetensors path. At TP2 the load time is about the same (37.3 s vs 34.9 s); the amount read grows with the TP size. Every yielded tensor was compared bit for bit against safetensors (no differences).
Collaborator
|
Hi @bobofang11235 could you clarify the accuracy result after this PR, and add a performance test script as well for reproduce. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
fastsafetensors_weights_iteratorcopies whole files to the device. A shardlarger than the memory left after model init fails with OOM: for
Qwen3.8-Flash-Next-PTPC-FP8-PLE-BF16 the first shard is 96 GB and only 78.48 GiB
is free per GPU at TP2 on MI308X (
Tried to allocate 95.37 GiB).Changes
SGLANG_FASTSAFETENSORS_MAX_BATCH_GIB: when set, load through fastsafetensors'ParallelLoader(requires fastsafetensors >= 0.5.0) withmax_batch_bytes, soeach file is read in chunks of that size.
default WORLD group the communicator (0.51 GiB per GPU) stays allocated for the
life of the process, because SGLang never uses that group otherwise, and the KV
cache shrinks accordingly.
SGLANG_FASTSAFETENSORS_ALL_LOCAL=1: every rank reads all shards itself insteadof receiving them by broadcast. On ROCm the broadcast path leaves about 0.47 GiB
of per-hardware-queue scratch allocated per GPU; with all_local the KV cache
matches the safetensors path.
Defaults are unchanged.
Accuracy
GSM8K on MI308X TP2, base branch
c996fcb1e, 3 runs each with the safetensors loader (base) and withthis change (chunked fastsafetensors, all_local), 5-shot in one chat message, thinking off,
temperature 0, 1,314 scored questions:
python3 -m sglang.test.run_eval --eval-name gsm8k --api chat --num-examples 1319 --max-tokens 1024 --chat-template-kwargs '{"enable_thinking": false}'The difference (+0.0015) is within run-to-run variation, as expected for bit-identical weights.
Results (MI308X, TP2,
--model-loader-extra-config '{"enable_gds": false}')Measured with the qwen4_exp load_weights fix applied, run back to back on the same GPUs:
With
FASTSAFETENSORS_ODIRECT=0(reads through the page cache) and the qwen4_exp expert-mappingindex, all_local loading takes 25.5 s.
Peak device memory while reading the 96 GB shard: 15.6 GiB with 8 GiB chunks.
Validation
broadcast and all_local mode: 149,318 tensors, 220.2 GiB, no differences.
all_local leaves nothing behind.
Notes
(3.0 vs 11.6 GB/s), so the measurements use
enable_gds: false.max_batch_bytesandall_localand yields bit-identical tensors, but this path is about 7x slowerwith it: the all_local iterator takes 48.7-49.5 s vs 6.3-6.6 s with 0.5.0 (same
session), and server weight loading on the fork takes 74.1 s vs 26.0-27.1 s, about
the same as safetensors (77.6 s). 0.3.x has no
max_batch_bytes.