Repository navigation
[recipe] Run Qwen3.5-9B instruct 16K on B200 - #363
Open
Micah Yong (micahtyong) wants to merge 1 commit into
Open
Micah Yong (micahtyong) wants to merge 1 commit into
Micah Yong (micahtyong) wants to merge 1 commit into
Conversation
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 18, 2026 00:57
Train on one B200 with no tensor parallelism and no activation recompute, and sample on B200 only with the DFlash drafter and settings the 32K recipe already uses. The recipes that inherited the old 16K layout (16k-dp2, 32k, 256k and qwen35-9b-lora-128k) now inherit a shared base with the same values, so their resolved configs are unchanged. The 32K recipe inherits the new 16K recipe and keeps its own trainer settings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
force-pushed
the
micah/qwen35-9b-instruct-16k-b200
branch
from
November 18, 2026 01:32
1f94986 to
2dad5d7
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
qwen35-9b-instruct-lora-16ktrained on eight H100s with tensor parallel size 8 and sampled on["B200", "H200"]. A 9B LoRA at 16K context fits on one B200, and splitting a model this small across eight GPUs mostly adds communication. This PR moves the recipe fully onto Blackwell:forward_backwardabout 27% faster than full recompute on the same GPU (0.73 s vs 1.00 s for one 16K-token row, 5.8 s vs 7.9 s for eight). The trainer keeps 6 clients and 6 LoRA slots, and the default 8 CPUs and 32 GiB.modal-labs/Qwen3.5-9B-DFlashdrafter and the sampler settings the 32K recipe already uses: 64 running requests,trtllm_mhaattention and FP32 recurrent state. It keeps this recipe's LoRA batching (8 adapters per batch, 256 loaded), since several clients share it. The H200 fallback is gone, because these settings run only on Blackwell.The images, canonical LoRA and raw export from #312 are unchanged.
Other recipes inherited the old 16K layout. Its values moved unchanged to a private base,
_qwen35_9b_instruct_lora.py, which16k-dp2,256kandqwen35-9b-lora-128know inherit. The 32K recipe now inherits the new 16K recipe and keeps its own trainer settings. Resolving all five recipes before and after shows that only the 16K recipe changes.A training checkpoint loads only on the trainer layout, image and adapter type that saved it. The new docs section "Older Qwen3.5-9B instruct 16K checkpoints" gives a variant for checkpoints from v0.1.7 or earlier.
Testing
All runs used the stock Tinker SDK against a staging frontend, with one sampler replica. The tuning runs below were on f1a6a3d, before #312 changed the images and adapter type.
On the PR head (main at 58e722a plus this commit):
forward_backward0.16–0.18 s. The median step was 16.0 s, up from 10.1 s before the rebase, because publishing each adapter now takes 7.8–11.6 s instead of 3–7 s. That comes from the new trainer image and adapter export, not the GPU layout; main's 8×H100 recipe wasn't measured on it (a, b).Tuning runs, before the rebase:
scripts/wandb_rl_example.py, 5 steps, two clients at once, with this recipe at f1a6a3d. Loss was finite on every step. Before is the v0.1.7 release check of this recipe on 8×H100, with the same script and two clients. In both, most of a step is publishing the adapter and sampling, not training. On 1866cf8, four clients sampling four adapters at once also passed. Their steps took 22–30 s, mostly because the trainer publishes each client's adapter in turn (2.6–4.4 s each);forward_backwardtook 0.14 s (W&B). There is no four-client run on the old layout to compare.compute_logprobsand the trainer'sforwardagree to a mean absolute difference of 0.003–0.004 (p99 0.10). The comparison was repeated after four SFT steps moved the adapter and it was published again: 0.0025–0.0027.forward_backwardin that stepW&B: sampler before, after, 1 client, after, 8 and 64 clients. RL example before a, before b, after a, after b. Parity. Trainer, recompute, trainer, no recompute.
Native MTP was also tried on the sampler at 128 running requests. It was slower than DFlash at 1 and 8 clients (MATH-500 ×4: 217 s and 38.7 s) and about the same from 32 clients up (W&B).
torch.compilestays on.TORCH_COMPILE_DISABLE=1, which the 32K recipe sets, shortened the firstforward_backwardfrom about 160–230 s to 30 s. But it made steady steps about 22% slower, and stalls on new batch shapes were longer without it. With compile on, a step with a new mix of row lengths still stalls for 6–11 s instead of about 3 s. That looks like the FLA kernel retuning that #333 fixed for the Megatron backend; the Miles LoRA backend doesn't pin those kernels. W&B: compile on, compile off.Not run: a B300 trainer or sampler, and a long RL run at 16K context.