Skip to content

[recipe] Run Qwen3.5-9B instruct 16K on B200 - #363

Open
Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/qwen35-9b-instruct-16k-b200
Open

Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/qwen35-9b-instruct-16k-b200

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

qwen35-9b-instruct-lora-16k trained on eight H100s with tensor parallel size 8 and sampled on ["B200", "H200"]. A 9B LoRA at 16K context fits on one B200, and splitting a model this small across eight GPUs mostly adds communication. This PR moves the recipe fully onto Blackwell:

  • Trainer: one B200, tensor parallel size 1, and no activation recompute. A full 16K-token micro-batch peaks at 148 GB of the B200's 179 GB. Turning recompute off makes forward_backward about 27% faster than full recompute on the same GPU (0.73 s vs 1.00 s for one 16K-token row, 5.8 s vs 7.9 s for eight). The trainer keeps 6 clients and 6 LoRA slots, and the default 8 CPUs and 32 GiB.
  • Sampler: B200 only, with the modal-labs/Qwen3.5-9B-DFlash drafter and the sampler settings the 32K recipe already uses: 64 running requests, trtllm_mha attention and FP32 recurrent state. It keeps this recipe's LoRA batching (8 adapters per batch, 256 loaded), since several clients share it. The H200 fallback is gone, because these settings run only on Blackwell.

The images, canonical LoRA and raw export from #312 are unchanged.

Other recipes inherited the old 16K layout. Its values moved unchanged to a private base, _qwen35_9b_instruct_lora.py, which 16k-dp2, 256k and qwen35-9b-lora-128k now inherit. The 32K recipe now inherits the new 16K recipe and keeps its own trainer settings. Resolving all five recipes before and after shows that only the 16K recipe changes.

A training checkpoint loads only on the trainer layout, image and adapter type that saved it. The new docs section "Older Qwen3.5-9B instruct 16K checkpoints" gives a variant for checkpoints from v0.1.7 or earlier.

Testing

All runs used the stock Tinker SDK against a staging frontend, with one sampler replica. The tuning runs below were on f1a6a3d, before #312 changed the images and adapter type.

On the PR head (main at 58e722a plus this commit):

  • MATH-500 ×4, one pass: 144 s at 1 client and 19.9 s at 64, the same as before the rebase (W&B).
  • RL example, two clients: loss finite on every step and forward_backward 0.16–0.18 s. The median step was 16.0 s, up from 10.1 s before the rebase, because publishing each adapter now takes 7.8–11.6 s instead of 3–7 s. That comes from the new trainer image and adapter export, not the GPU layout; main's 8×H100 recipe wasn't measured on it (a, b).
  • Logprob parity on the canonical adapters: mean absolute difference 0.003–0.0035 (p99 0.09), and 0.0027–0.0029 after four SFT steps and a republish (W&B).

Tuning runs, before the rebase:

  • Sampler: the small-model eval sweep: 128 IFEval prompts (greedy, 1,280 max tokens) and 64 MATH-500 problems × 4 samples (T 0.6, 2,048 max tokens), two passes each. "Before" is main's recipe at f1a6a3d, and it got a B200. The DFlash runs used the same recipe values at f1a6a3d as config overrides. The 1-client column is from the run with 24 CPUs and 128 GiB, as in the 32K recipe. The other columns are from the run with this PR's default 8 CPUs and 32 GiB. Its times matched the larger run within 4%.
  • Trainer: scripts/wandb_rl_example.py, 5 steps, two clients at once, with this recipe at f1a6a3d. Loss was finite on every step. Before is the v0.1.7 release check of this recipe on 8×H100, with the same script and two clients. In both, most of a step is publishing the adapter and sampling, not training. On 1866cf8, four clients sampling four adapters at once also passed. Their steps took 22–30 s, mostly because the trainer publishes each client's adapter in turn (2.6–4.4 s each); forward_backward took 0.14 s (W&B). There is no four-client run on the old layout to compare.
  • Logprob parity: 32 sequences sampled at T 1 with DFlash on. Their sampled logprobs, compute_logprobs and the trainer's forward agree to a mean absolute difference of 0.003–0.004 (p99 0.10). The comparison was repeated after four SFT steps moved the adapter and it was published again: 0.0025–0.0027.
Before After
IFEval, 1 client 294 s 187 s
IFEval, 64 clients 14.4 s 11.1 s
MATH-500 ×4, 1 client 450 s 140 s
MATH-500 ×4, 8 clients 76.6 s 30.5 s
MATH-500 ×4, 64 clients 29.5 s 18.8 s
RL example step, median of steps 1–4 9.3 s 10.1 s
Trainer forward_backward in that step 0.53 s 0.23 s
Trainer GPUs, list $/h 8 × H100, $31.60 1 × B200, $6.25

W&B: sampler before, after, 1 client, after, 8 and 64 clients. RL example before a, before b, after a, after b. Parity. Trainer, recompute, trainer, no recompute.

Native MTP was also tried on the sampler at 128 running requests. It was slower than DFlash at 1 and 8 clients (MATH-500 ×4: 217 s and 38.7 s) and about the same from 32 clients up (W&B).

torch.compile stays on. TORCH_COMPILE_DISABLE=1, which the 32K recipe sets, shortened the first forward_backward from about 160–230 s to 30 s. But it made steady steps about 22% slower, and stalls on new batch shapes were longer without it. With compile on, a step with a new mix of row lengths still stalls for 6–11 s instead of about 3 s. That looks like the FLA kernel retuning that #333 fixed for the Megatron backend; the Miles LoRA backend doesn't pin those kernels. W&B: compile on, compile off.

Not run: a B300 trainer or sampler, and a long RL run at 16K context.

Train on one B200 with no tensor parallelism and no activation
recompute, and sample on B200 only with the DFlash drafter and settings
the 32K recipe already uses.

The recipes that inherited the old 16K layout (16k-dp2, 32k, 256k and
qwen35-9b-lora-128k) now inherit a shared base with the same values, so
their resolved configs are unchanged. The 32K recipe inherits the new
16K recipe and keeps its own trainer settings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) force-pushed the micah/qwen35-9b-instruct-16k-b200 branch from 1f94986 to 2dad5d7 Compare November 18, 2026 01:32

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant