Skip to content

[CI] tests: Let test_hicache_storage size its host cache to the runner - #44729

Merged
xiezhq-hermann merged 1 commit into
mainfrom
oss-sync-hicache-storage-host-size
Oct 16, 2026
Merged

xiezhq-hermann merged 1 commit into
mainfrom
oss-sync-hicache-storage-host-size

Conversation

@metamergebot

@metamergebot metamergebot commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Motivation

base-b-test-1-gpu-small (7) fails on PRs whenever it lands on a smaller 5090 runner: test/registered/hicache/test_hicache_storage.py launches its server with --hicache-size 100, a fixed 100 GB pinned host pool, and on runners with roughly 51-81 GB of host memory the server is OOM-killed at launch:

CI_OFFLINE: Offline launch failed (Server process exited with code -9), retrying with online mode...
Exception: Server process exited with code -9. Check server logs for errors.
✗ FAILED: .../test/registered/hicache/test_hicache_storage.py returned exit code 5

The same shard passes when it draws a larger runner, so affected PRs only get through by rerunning until they do. It hit four unrelated PRs on 2026-10-16 (#44583, #44533, #44613, #44614).

Modifications

On CUDA the test no longer passes --hicache-size. With neither --hicache-size nor --hicache-ratio set, the server uses the default host/device ratio and --hicache-host-memory-fraction (0.8 by default) reduces it until the host pools fit the machine's available memory, including the cgroup limit. The test still exercises the hierarchical cache with the file storage backend and the same MMLU threshold. ROCm keeps its explicit --hicache-size 200.

Test plan

  • black, isort, ruff check --select=F401,F821,F823,UP037 clean on the changed file.
  • Not run locally: the test's model (meta-llama/Llama-3.1-8B-Instruct) is gated on the available machine, and its environment predates main's kernel requirements. This PR's own base-b-test-1-gpu-small (7) runs the test on the affected 5090 runners and is the validation.

CI States

Latest PR Test (Base): 🚫 Run #38891355502
Latest PR Test (Extra): ❌ Run #38891355100
Latest PR Test (AMD ROCm 10): 🚫 Run #38891355239

test_hicache_storage pinned a 100 GB HiCache host pool on CUDA. On the
1-gpu-small 5090 runners with 51-81 GB of host memory the server is
OOM-killed at launch (exit -9), failing base-b-test-1-gpu-small (7)
whenever a PR draws one of them. Without --hicache-size the default
ratio applies and --hicache-host-memory-fraction reduces it to fit the
machine's memory and cgroup limit. ROCm keeps its explicit 200 GB.
@metamergebot metamergebot added hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR labels Oct 16, 2026
@metamergebot metamergebot added the bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) label Oct 16, 2026
@metamergebot

Copy link
Copy Markdown
Collaborator Author

/rerun-test registered/hicache/test_hicache_storage.py

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/hicache/test_hicache_storage.py

@metamergebot

Copy link
Copy Markdown
Collaborator Author

Verified with /rerun-test registered/hicache/test_hicache_storage.py on this head (ec5ba3641e), run 38950074179: passed on 5090-vast-es-0, one of the ~52 GB 1-gpu-small runners where the fixed 100 GB pool fails today.

HiCache memory headroom: host 51.8 GiB, cgroup 51.5 GiB
Allocating kv hierarchical KV host pool: 106304 tokens, 13.93 GB host memory.
mmlu_score=0.66796875   (threshold 0.64)

On main the same runner class fails with Not enough host memory available. Requesting 100.00 GB but only have 46.12 GB free (seen this week on #44622, #44592 and others in base-b-test-1-gpu-small (7)).

@xiezhq-hermann
xiezhq-hermann merged commit a31a8c2 into main Oct 16, 2026
442 of 538 checks passed
@xiezhq-hermann
xiezhq-hermann deleted the oss-sync-hicache-storage-host-size branch October 16, 2026 20:38
detain pushed a commit to detain/sglang that referenced this pull request Oct 19, 2026
sgl-project#44729)

Co-authored-by: metamergebot <metamergebot@users.noreply.github.com>
(cherry picked from commit a31a8c2)
(cherry picked from commit 32d9dfb8b3392ddf32608f902d747be586ac3a9c)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bypass-fail-fast CI: a failing job no longer aborts its siblings (lint still gates) hicache Hierarchical Caching for SGLang run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants