Skip to content

[GLM-5.3 Flash 256K] Forward compressed forward_backward bodies to the trainer - #307

Open
Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/fwdbwd-compressed-forward
Open

Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/fwdbwd-compressed-forward

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A forward_backward request goes from the Tinker client to the frontend, then from the frontend to the trainer. Tinker zstd-compresses the request body. The frontend used to decompress it, read the model and sequence IDs, and send the decompressed bytes on to the trainer with httpx's default 30 s timeout.

That breaks with long sequences and expert-routing replay. Routes are int64 values for every position, layer and expert slot, so a 256K-token GLM-5.3 Flash request is about 130 MB compressed and 760 MB decompressed. The trainer often took more than 30 s to take in that body, so the hop timed out: the client got 503s and retried, and the trainer logged ClientDisconnect.

The frontend now still reads the IDs, then forwards the original compressed bytes with their Content-Encoding. The engine decompresses them off the event loop. This hop also gets the Tinker SDK's own 60 s request timeout instead of 30 s. Smaller requests behave the same, apart from crossing the hop compressed.

E2E results

Setup: glm53-flash-lora-256k (#308) on one 8×B300 trainer node. The client is the stock Tinker SDK with spindle.sample_with_replay(return_routed_experts=True). Each step samples a 262,080-token prompt, then trains on prompt + completion (262,141 tokens) with the captured routes.

Before WandB / After WandB

262,141-token replay request Before After
Frontend forward_backward submit 503 at 42 s, 503 at 43 s, then 200 at 46 s 200 at 23 s, 200 at 21 s

The frontend decompressed each forward_backward request only to read its
model and sequence IDs, then sent the decompressed bytes to the trainer under
httpx's default 30 s timeout. A 256K-token request with expert-routing replay
is about 130 MB compressed and 760 MB decompressed; the trainer took over 30 s
to ingest it, so attempts failed with 503s. Forward the original bytes with
their Content-Encoding, decompress them in the engine off the event loop, and
give this hop the Tinker SDK's 60 s request timeout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant