Repository navigation
[Minor Fix] Fix llava modalities issue for single-image - #1402
Merged
Merged
Conversation
Contributor
Author
This was referenced Sep 12, 2024
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
…ce probe, ring buffers and one indexed host write per batch for KV and draft pages, preadv
Boots xsn129-132: the D group read a store prefix at 250-430 pages/s per
rank (11k pages -> 13 s, past the 13.5 s reap budget: completed=2680 of
11806 on xsn132), so every leg-2 request parked, the box idled, and the
W1b abort had to end the drain. py-spy on D TP0 during the in-flip read
of xsn132 (profile in the job dir, 2663 samples):
prefetch issuer thread: 691/710 samples in batch_exists_v2, 533 of them
in os.path.exists -- the probe of one stem was exists(sharded) +
exists(flat) + exists(sharded) + getsize, and an 11k prefix probes
33k stems (KV, mamba, draft). Measured on the store: 11.5 us a stem
as written, 2.7 us as one os.stat.
prefetch IO thread: ~half in set_from_flat_data_page (2*layer_num
strided slivers per 32 KiB page), the rest split between a fresh
pinned get_dummy_flat_data_page() per page (the sgl-project#720 ring served only
the extra-pool route, never the KV/draft pages), read_extents' pread
alloc+copy, _existing_path's second exists, and the evictor touch.
The disk itself: 6-7 us a warm 32 KiB page (readinto/preadv measured).
Changes, all on the same read path:
hicache_storage._stat_stem: ONE os.stat answers presence, path and
size; the legacy flat layout is consulted only when the top directory
holds flat .bin files (decided once per backend). _stem_exists,
_existing_path and _stem_readable are its three views.
cache_controller: _generic_page_get and _draft_page_get_generic borrow
their targets from the backend's read ring (_borrow_read_pages, module
level so the harness doubles that bind curated methods keep working);
a draft miss zeroes its borrowed buffer (sgl-project#993 stays: the zero page is
the miss); the served prefix of a batch is written with ONE indexed
copy (_set_host_pages -> set_from_flat_data_pages).
pool_host/mha.py: set_from_flat_data_pages for layer_first/page_first
(one index assignment over the token axis); base.py default loop;
memory_pool_host's bound-tier wrapper keeps the per-index stray check.
canonical_page_store.read_extents: preadv straight into the target.
Tests: test_read_ring_prefix_routes_1402 (6: ring borrow/return, dirty
buffer -> zero page, no-ring fallback, raised read returns buffers,
batched write == per-page loop for both layouts x page_size 1/2, served
prefix written once and stops at the miss) + sharding_558, canonical
1233/706, mem_pool_host, stray_718, draftkv, reissue_939, 1157, 869b,
905, 937, 966, 1063, 861, 0828, 1324, 1401: 221 passed, 7 failed -- all
seven red at the tip without this change (3 mem_pool_host + 1 in 905 +
2 in 1063 + the 0828 text scan that trips on a 2026-09-07 comment).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
… page by default (SGLANG_HICACHE_CANONICAL_FSYNC=1 restores it) py-spy on P PP0 during xsn132's in-flip window: the backup thread was busy for the whole profile (341 samples, _page_backup -> set -> _set_canonical_slice -> write_extents), the write_back sweep before the flip drained 1-2 nodes a second, and the front held the P->D flip for ~10 s on hicache_backup. Per 32 KiB page the completing stage ran lock + fstat + marker read + pwrite + FSYNC + rename + unlink on ZFS. The store is the carrier between the two groups of ONE host: the reader finds the renamed blob through the same page cache, so the fsync bought nothing a flip could use, and a crash loses the boot's store either way (upstream's file backend never fsyncs a page). Rename-on-complete is unchanged -- a reader still never sees a partial blob. The explicit `fsync=` argument keeps winning over the env. Tests: test_canonical_fsync_default_1402 (3) + canonical_page_store_706, kv_page_704b, kv_major_1233, mamba_blob_706, layer_cut_706, gdn_931: 93 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
…-off per syscall: batched page I/O in C, one release per batch, and a 0.2 ms switch interval Boot xsn133 (db4bce6, 2026-09-15 22:13Z) with the one-stat probe, the ring buffers and the batched host write: D's leg-2 reads went from 250-430 to 350-650 pages/s -- still far under the 11k pages a 13.5 s reap budget needs, so both 11k requests were reaped, re-issued from zero (completed 7936, 7040, 5120, ... per re-issue), ADMISSION-WEDGE 69 lines, epoch 9 drain 120 s, W1b again. py-spy on D TP0: the IO thread's time was read_extents 38 %, _stat_stem 22 %, set_from_flat_data_pages 20 %, utime 9 % -- yet the same syscalls cost 7 us a page on the desk. The cause, measured (gil_probe against the boot's own store, a read of six syscalls per page beside one CPU-bound Python thread in the process): alone 13 us/page busy thread, switch 5 ms 20 591 us/page busy thread, switch 1 ms 4 522 us/page busy thread, switch 0.2 ms 1 073 us/page Every os.* call releases the GIL and waits up to the switch interval to get it back; the scheduler's main thread is busy, so a 32 KiB page paid six hand-offs, the draft page six more, and the presence probe of an 11k prefix 33k of them. Two fixes, independent: storage/file/pageio.c + pageio.py: hicache_stat_sizes and hicache_read_pages loop over the same syscalls in C, called through ctypes = ONE GIL release per batch (128 pages, or all 33k probe stems). Built on first use with gcc into the temp dir, keyed by the source hash; unavailable -> the Python path, reason logged once; SGLANG_HICACHE_PAGEIO=0 disables it. HiCacheFile.batch_get routes canonical-window keys through it (the evictor's LRU is updated with mtime=False because the helper bumped the inode through its fd); _collect_existing_component_keys probes through _stat_stems / _readable_stems, the same readability rule, one call. run_scheduler_process: SGLANG_GIL_SWITCH_INTERVAL_MS sets sys.setswitchinterval; the launcher defaults it to 0.2 ms so every remaining Python-level syscall on the prefetch threads waits at most 0.2 ms instead of 5. test_hicache_sharding_558's fsync-failure case now asks for the fsync it tests (fsync=True; the default went off in 781bbf4). Tests: test_pageio_batch_read_1402 (5: sizes, extents+touch, statuses, env switch, batch beside a busy thread >= 5x a per-syscall read) + 0828 roundtrip, sharding_558, canonical 706/1233, 1157, 869b, layer_cut, mamba_gates, canonical_key, read_ring_1402, fsync_1402, 1324: 214 passed; the 0828 text scan stays red on its 2026-09-07 comment as at the tip.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
… the publish sweep before a flip runs one GIL release per batch Boot xsn134: with the read path fixed, the largest remaining flip term was the P->D drain of 10.5 s at epoch 8 (P-IDLE-VERDICT blockers = hicache_backup) -- the write_back publish sweep of three 11k prefixes, one canonical write per 32 KiB page per PP stage. py-spy on P PP0 had the backup thread in write_extents (43 %), get_data_page (17 %) and the marker unlink (13 %) for the whole in-flip window: lock, fstat, marker read, ftruncate, pwrite, marker write / rename + unlink -- every one a GIL hand-off against the busy main thread. pageio.c hicache_write_pages: the SAME protocol as write_extents and _open_part_locked (part file + flock EX with the inode re-check and 8 retries, marker sidecar "SGL706\x02" | u64 total | u16 count | (u64,u64) LE, foreign marker -> reset, ftruncate to width, pwrite the extents, merge coverage, marker on partial / optional fsync + rename + unlink on completion), page by page under one release of the GIL. Status per page: 0 published, 1 partial, 2 already complete, 3 shape refused, 4 io/lock. HiCacheFile.batch_set: canonical-window keys go through it in one call (presence probe for the batch through _stat_stems first; the evictor's reserve/commit/abort and the metadata cache per key exactly as before); other keys and shape refusals take set()/_set_canonical_slice unchanged. write_extents in Python stays the single-page reference implementation. Tests: test_pageio_batch_write_1402 (4): C partial -> marker byte-identical to _encode_marker and decodable by _decode_marker, C completes -> bytes, no marker/part, third write = already complete; Python starts / C completes and C starts / Python completes, readable by read_extents; a foreign part without marker is reset, not reused; a batch mixing complete and new. Plus pageio read (5), 0828 roundtrip, sharding 558, canonical 706/1233, 1157, 869b, layer_cut, 1063: 155 passed; the same 3 sockel reds as at the tip (0828 text scan, 1063 x2).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 15, 2026
…st pool in one gather per batch Boot xsn137 (4aecbb1): with the batched C write the P->D drain at epoch 8 fell from 10.5 s (xsn134) to 4.9 s, and py-spy on P PP0 had the backup thread at write_pages 41 % / get_data_page 36 % -- the per-page strided slice + flatten (2 x layer_num slivers) is the read-side twin of the write the set_from_flat_data_pages batch already removed. pool_host/mha.py get_data_pages: one indexed gather + one permute for layer_first / page_first, N contiguous views back; base.py default loop; memory_pool_host's bound-tier wrapper keeps the per-index stray check; _generic_page_set uses it where the pool offers it. Tests: read_ring_1402 (+1: batched read == per-page loop for both layouts x page_size 1/2, contiguous views) + 0828 roundtrip, stray_718, asymmetric pool, mem_pool_host, pageio write: 39 passed, the 4 reds are the tip's (0828 text scan, mem_pool_host x3).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 16, 2026
… mamba blobs) take the batched C path too Boot xsn138 (c0252e7): the publish sweep before P->D still drained 3.9 s; py-spy on P PP0 had the backup thread in batch_set_v2 -> _batch_io_v2 -> _write_page -> get_data_page + set -- one Python canonical write per draft page, beside the batched KV pages. batch_set is now all() over _batch_set_each (per-key results), and batch_set_v2 gathers a transfer's pages (get_data_pages where the pool offers it) and writes them through _batch_set_each in one C call per transfer. Tests: pageio write, 0828 roundtrip, 869b, draftkv canonical 1233, sharding 558, mamba blob 706, 1157, nixl storage, read_ring 1402, 1063: 112 passed, the 3 reds are the tip's text scans (0828 x1, 1063 x2).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 16, 2026
…per batch (get_data_pages), the last per-page host read on the publish sweep's thread (py-spy P PP0 xsn139: 16 %). Tests: read_ring_1402 (+1), draft_cold_admission_861, draft_hicache_binding_861: 43 passed.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Sep 26, 2026
…g je Prefetch -- der letzte Draft-Tier-Rest auf NF-D Nutzer-Order 24.09. 14:15Z: "schreiben wir in D auch draft context in den hicache? das muesste raus, weil draft ja keinen hicacheplatz mehr bekommt." Stand auf d1c7094: der Tier-Fix selbst ist schon drin (27B 9eef037, NF-Pick H53 b28e192 + ac26f23 + 53d2685). Metall fnFL2h91v1 (39fd662): Front 'WEG2 HICACHE-DRAFT-TIER: off (auto: group P has no draft producer)', alle drei D-Raenge '... off ... at boot registration', 0x 'HiCache draft KV registered', 0x 'sgl-project#993 draft L3 READ', Admission 'draft tier armed=False', Ledger arena=23.75 statt 25.59 (x158). Rest: die fnFL2x22-Formabstimmung (_agree_claim_form, ein skalares MAX all_reduce ueber die Prefetch-Gruppen je Prefetch-Operation, vor dem Claim-MIN) laeuft weiter, obwohl die gepackte Form nur fuer den Draft-Claim existiert (draft_tier_armed('admission') oder der Solo-Shadow-Marker). Unter off hat kein Rang eines von beiden: h91v1 D 24/24 'local_packed=False agreed_packed=False' auf TP0/TP1/TP2, x169 bis n=1280 je Boot -- ein Kollektiv je Prefetch ohne Nutzen, mit 120-s-Schranke als Haltepunkt. Fix (cache_controller, modulweit wie sgl-project#1402 verlangt): claim_form_scalar_by_switch -- unter SGLANG_WEG2_HICACHE_DRAFT_TIER=off (vom Launcher aufgeloest in BEIDE Gruppen-Envs geschrieben, eine Prefetch-Gruppe liegt immer in einer Launch-Gruppe -> ein Wert je Gruppe) gibt _agree_claim_form ohne Kollektiv 'skalar' zurueck; der Claim-Reduce ist das skalare MIN wie bisher -> gleicher Claim, eine Kollektive statt zwei je Operation. Einmal je Controller die Zeile 'H51 PREFETCH-CLAIM-FORM scalar by switch (SGLANG_WEG2_HICACHE_DRAFT_TIER=off) ...' statt der '#xsn392b PREFETCH-CLAIM-FORM'-Zeilen. Ein Rang, der unter off 'packed' antwortet, widerspricht der Konstruktion (Registrierung kehrt vor set_draft_kv_pool UND vor dem Shadow-Marker zurueck, der Cutover entwaffnet) und waere mit seinen Peers ungleich gepaart -> benannter Gruppen-STOP (Weg2DraftDisagree 'H51 PREFETCH-CLAIM-FORM' -> _stop_group_from_thread). Unset/auto/on: die Abstimmung wie bisher, byte-gleich. Kein neuer Schalter. 27B-Linie unberuehrt (anderer Baum); H91d (nicht auf rc2.1l) liest nur hicache_draft_tier_off und nutzt den Tier nicht. Tests (test_weg2_hicache_draft_tier_nf_h51.py, Abschnitt H51 rc2.1l, 9): rot auf d1c7094 (4): Gruppe unter off ohne Form-Kollektiv (TP0+2 Shadows), eine benannte Zeile, echter prefetch_thread_func-Lauf nur MIN je Operation bei gleichem Claim 4032, packed-Rang unter off = benannter STOP. Gruen vorher und nachher (5): on/unset/auto behalten das MAX, Gruppe P ohne off stimmt ab, ohne Sync-Gruppe keine Kollektive und keine Zeile.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Motivation
As described in #1400 , LLaVA is performing worse under single-image evaluation
Modifications
Fix the condition issue when forwarding single-image using llava. Adapt the new modalities parameters with single-image
Checklist