Skip to content

[Minor Fix] Fix llava modalities issue for single-image - #1402

Merged
merrymercy merged 3 commits into
sgl-project:mainfrom
EvolvingLMMs-Lab:fix_llava
Sep 12, 2024
Merged

merrymercy merged 3 commits into
sgl-project:mainfrom
EvolvingLMMs-Lab:fix_llava

Conversation

@kcz358

@kcz358 kcz358 commented Sep 12, 2024

Copy link
Copy Markdown
Contributor

Motivation

As described in #1400 , LLaVA is performing worse under single-image evaluation

Modifications

Fix the condition issue when forwarding single-image using llava. Adapt the new modalities parameters with single-image

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@kcz358

kcz358 commented Sep 12, 2024 •

Copy link
Copy Markdown
Contributor Author

Alignment result

AI2D

image

MME

image

@merrymercy
merrymercy merged commit 8234e66 into sgl-project:main Sep 12, 2024
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 15, 2026
…ce probe, ring buffers and one indexed host write per batch for KV and draft pages, preadv

Boots xsn129-132: the D group read a store prefix at 250-430 pages/s per
rank (11k pages -> 13 s, past the 13.5 s reap budget: completed=2680 of
11806 on xsn132), so every leg-2 request parked, the box idled, and the
W1b abort had to end the drain. py-spy on D TP0 during the in-flip read
of xsn132 (profile in the job dir, 2663 samples):

  prefetch issuer thread: 691/710 samples in batch_exists_v2, 533 of them
    in os.path.exists -- the probe of one stem was exists(sharded) +
    exists(flat) + exists(sharded) + getsize, and an 11k prefix probes
    33k stems (KV, mamba, draft). Measured on the store: 11.5 us a stem
    as written, 2.7 us as one os.stat.
  prefetch IO thread: ~half in set_from_flat_data_page (2*layer_num
    strided slivers per 32 KiB page), the rest split between a fresh
    pinned get_dummy_flat_data_page() per page (the sgl-project#720 ring served only
    the extra-pool route, never the KV/draft pages), read_extents' pread
    alloc+copy, _existing_path's second exists, and the evictor touch.
  The disk itself: 6-7 us a warm 32 KiB page (readinto/preadv measured).

Changes, all on the same read path:
  hicache_storage._stat_stem: ONE os.stat answers presence, path and
    size; the legacy flat layout is consulted only when the top directory
    holds flat .bin files (decided once per backend). _stem_exists,
    _existing_path and _stem_readable are its three views.
  cache_controller: _generic_page_get and _draft_page_get_generic borrow
    their targets from the backend's read ring (_borrow_read_pages, module
    level so the harness doubles that bind curated methods keep working);
    a draft miss zeroes its borrowed buffer (sgl-project#993 stays: the zero page is
    the miss); the served prefix of a batch is written with ONE indexed
    copy (_set_host_pages -> set_from_flat_data_pages).
  pool_host/mha.py: set_from_flat_data_pages for layer_first/page_first
    (one index assignment over the token axis); base.py default loop;
    memory_pool_host's bound-tier wrapper keeps the per-index stray check.
  canonical_page_store.read_extents: preadv straight into the target.

Tests: test_read_ring_prefix_routes_1402 (6: ring borrow/return, dirty
buffer -> zero page, no-ring fallback, raised read returns buffers,
batched write == per-page loop for both layouts x page_size 1/2, served
prefix written once and stops at the miss) + sharding_558, canonical
1233/706, mem_pool_host, stray_718, draftkv, reissue_939, 1157, 869b,
905, 937, 966, 1063, 861, 0828, 1324, 1401: 221 passed, 7 failed -- all
seven red at the tip without this change (3 mem_pool_host + 1 in 905 +
2 in 1063 + the 0828 text scan that trips on a 2026-09-07 comment).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 15, 2026
… page by default (SGLANG_HICACHE_CANONICAL_FSYNC=1 restores it)

py-spy on P PP0 during xsn132's in-flip window: the backup thread was
busy for the whole profile (341 samples, _page_backup -> set ->
_set_canonical_slice -> write_extents), the write_back sweep before the
flip drained 1-2 nodes a second, and the front held the P->D flip for
~10 s on hicache_backup. Per 32 KiB page the completing stage ran
lock + fstat + marker read + pwrite + FSYNC + rename + unlink on ZFS.

The store is the carrier between the two groups of ONE host: the reader
finds the renamed blob through the same page cache, so the fsync bought
nothing a flip could use, and a crash loses the boot's store either way
(upstream's file backend never fsyncs a page). Rename-on-complete is
unchanged -- a reader still never sees a partial blob. The explicit
`fsync=` argument keeps winning over the env.

Tests: test_canonical_fsync_default_1402 (3) + canonical_page_store_706,
kv_page_704b, kv_major_1233, mamba_blob_706, layer_cut_706, gdn_931:
93 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 15, 2026
…-off per syscall: batched page I/O in C, one release per batch, and a 0.2 ms switch interval

Boot xsn133 (db4bce6, 2026-09-15 22:13Z) with the one-stat probe, the
ring buffers and the batched host write: D's leg-2 reads went from
250-430 to 350-650 pages/s -- still far under the 11k pages a 13.5 s reap
budget needs, so both 11k requests were reaped, re-issued from zero
(completed 7936, 7040, 5120, ... per re-issue), ADMISSION-WEDGE 69 lines,
epoch 9 drain 120 s, W1b again. py-spy on D TP0: the IO thread's time was
read_extents 38 %, _stat_stem 22 %, set_from_flat_data_pages 20 %,
utime 9 % -- yet the same syscalls cost 7 us a page on the desk.

The cause, measured (gil_probe against the boot's own store, a read of six
syscalls per page beside one CPU-bound Python thread in the process):

    alone                         13 us/page
    busy thread, switch 5 ms  20 591 us/page
    busy thread, switch 1 ms   4 522 us/page
    busy thread, switch 0.2 ms 1 073 us/page

Every os.* call releases the GIL and waits up to the switch interval to get
it back; the scheduler's main thread is busy, so a 32 KiB page paid six
hand-offs, the draft page six more, and the presence probe of an 11k
prefix 33k of them.

Two fixes, independent:
  storage/file/pageio.c + pageio.py: hicache_stat_sizes and
    hicache_read_pages loop over the same syscalls in C, called through
    ctypes = ONE GIL release per batch (128 pages, or all 33k probe
    stems). Built on first use with gcc into the temp dir, keyed by the
    source hash; unavailable -> the Python path, reason logged once;
    SGLANG_HICACHE_PAGEIO=0 disables it. HiCacheFile.batch_get routes
    canonical-window keys through it (the evictor's LRU is updated with
    mtime=False because the helper bumped the inode through its fd);
    _collect_existing_component_keys probes through _stat_stems /
    _readable_stems, the same readability rule, one call.
  run_scheduler_process: SGLANG_GIL_SWITCH_INTERVAL_MS sets
    sys.setswitchinterval; the launcher defaults it to 0.2 ms so every
    remaining Python-level syscall on the prefetch threads waits at most
    0.2 ms instead of 5.

test_hicache_sharding_558's fsync-failure case now asks for the fsync it
tests (fsync=True; the default went off in 781bbf4).

Tests: test_pageio_batch_read_1402 (5: sizes, extents+touch, statuses,
env switch, batch beside a busy thread >= 5x a per-syscall read) +
0828 roundtrip, sharding_558, canonical 706/1233, 1157, 869b, layer_cut,
mamba_gates, canonical_key, read_ring_1402, fsync_1402, 1324: 214 passed;
the 0828 text scan stays red on its 2026-09-07 comment as at the tip.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 15, 2026
… the publish sweep before a flip runs one GIL release per batch

Boot xsn134: with the read path fixed, the largest remaining flip term
was the P->D drain of 10.5 s at epoch 8 (P-IDLE-VERDICT blockers =
hicache_backup) -- the write_back publish sweep of three 11k prefixes,
one canonical write per 32 KiB page per PP stage. py-spy on P PP0 had the
backup thread in write_extents (43 %), get_data_page (17 %) and the
marker unlink (13 %) for the whole in-flip window: lock, fstat, marker
read, ftruncate, pwrite, marker write / rename + unlink -- every one a
GIL hand-off against the busy main thread.

pageio.c hicache_write_pages: the SAME protocol as write_extents and
_open_part_locked (part file + flock EX with the inode re-check and 8
retries, marker sidecar "SGL706\x02" | u64 total | u16 count | (u64,u64)
LE, foreign marker -> reset, ftruncate to width, pwrite the extents,
merge coverage, marker on partial / optional fsync + rename + unlink on
completion), page by page under one release of the GIL. Status per page:
0 published, 1 partial, 2 already complete, 3 shape refused, 4 io/lock.

HiCacheFile.batch_set: canonical-window keys go through it in one call
(presence probe for the batch through _stat_stems first; the evictor's
reserve/commit/abort and the metadata cache per key exactly as before);
other keys and shape refusals take set()/_set_canonical_slice unchanged.
write_extents in Python stays the single-page reference implementation.

Tests: test_pageio_batch_write_1402 (4): C partial -> marker
byte-identical to _encode_marker and decodable by _decode_marker, C
completes -> bytes, no marker/part, third write = already complete;
Python starts / C completes and C starts / Python completes, readable by
read_extents; a foreign part without marker is reset, not reused; a batch
mixing complete and new. Plus pageio read (5), 0828 roundtrip, sharding
558, canonical 706/1233, 1157, 869b, layer_cut, 1063: 155 passed; the
same 3 sockel reds as at the tip (0828 text scan, 1063 x2).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 15, 2026
…st pool in one gather per batch

Boot xsn137 (4aecbb1): with the batched C write the P->D drain at
epoch 8 fell from 10.5 s (xsn134) to 4.9 s, and py-spy on P PP0 had the
backup thread at write_pages 41 % / get_data_page 36 % -- the per-page
strided slice + flatten (2 x layer_num slivers) is the read-side twin of
the write the set_from_flat_data_pages batch already removed.

pool_host/mha.py get_data_pages: one indexed gather + one permute for
layer_first / page_first, N contiguous views back; base.py default loop;
memory_pool_host's bound-tier wrapper keeps the per-index stray check;
_generic_page_set uses it where the pool offers it.

Tests: read_ring_1402 (+1: batched read == per-page loop for both
layouts x page_size 1/2, contiguous views) + 0828 roundtrip, stray_718,
asymmetric pool, mem_pool_host, pageio write: 39 passed, the 4 reds are
the tip's (0828 text scan, mem_pool_host x3).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 16, 2026
… mamba blobs) take the batched C path too

Boot xsn138 (c0252e7): the publish sweep before P->D still drained
3.9 s; py-spy on P PP0 had the backup thread in batch_set_v2 ->
_batch_io_v2 -> _write_page -> get_data_page + set -- one Python
canonical write per draft page, beside the batched KV pages. batch_set is
now all() over _batch_set_each (per-key results), and batch_set_v2
gathers a transfer's pages (get_data_pages where the pool offers it) and
writes them through _batch_set_each in one C call per transfer.

Tests: pageio write, 0828 roundtrip, 869b, draftkv canonical 1233,
sharding 558, mamba blob 706, 1157, nixl storage, read_ring 1402, 1063:
112 passed, the 3 reds are the tip's text scans (0828 x1, 1063 x2).
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 16, 2026
…per batch (get_data_pages), the last per-page host read on the publish sweep's thread (py-spy P PP0 xsn139: 16 %). Tests: read_ring_1402 (+1), draft_cold_admission_861, draft_hicache_binding_861: 43 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Sep 26, 2026
…g je Prefetch -- der letzte Draft-Tier-Rest auf NF-D

Nutzer-Order 24.09. 14:15Z: "schreiben wir in D auch draft context in den
hicache? das muesste raus, weil draft ja keinen hicacheplatz mehr bekommt."

Stand auf d1c7094: der Tier-Fix selbst ist schon drin (27B 9eef037,
NF-Pick H53 b28e192 + ac26f23 + 53d2685). Metall fnFL2h91v1
(39fd662): Front 'WEG2 HICACHE-DRAFT-TIER: off (auto: group P has no draft
producer)', alle drei D-Raenge '... off ... at boot registration', 0x 'HiCache
draft KV registered', 0x 'sgl-project#993 draft L3 READ', Admission 'draft tier
armed=False', Ledger arena=23.75 statt 25.59 (x158).

Rest: die fnFL2x22-Formabstimmung (_agree_claim_form, ein skalares MAX
all_reduce ueber die Prefetch-Gruppen je Prefetch-Operation, vor dem
Claim-MIN) laeuft weiter, obwohl die gepackte Form nur fuer den Draft-Claim
existiert (draft_tier_armed('admission') oder der Solo-Shadow-Marker). Unter
off hat kein Rang eines von beiden: h91v1 D 24/24 'local_packed=False
agreed_packed=False' auf TP0/TP1/TP2, x169 bis n=1280 je Boot -- ein
Kollektiv je Prefetch ohne Nutzen, mit 120-s-Schranke als Haltepunkt.

Fix (cache_controller, modulweit wie sgl-project#1402 verlangt):
claim_form_scalar_by_switch -- unter SGLANG_WEG2_HICACHE_DRAFT_TIER=off
(vom Launcher aufgeloest in BEIDE Gruppen-Envs geschrieben, eine
Prefetch-Gruppe liegt immer in einer Launch-Gruppe -> ein Wert je Gruppe)
gibt _agree_claim_form ohne Kollektiv 'skalar' zurueck; der Claim-Reduce
ist das skalare MIN wie bisher -> gleicher Claim, eine Kollektive statt
zwei je Operation. Einmal je Controller die Zeile 'H51 PREFETCH-CLAIM-FORM
scalar by switch (SGLANG_WEG2_HICACHE_DRAFT_TIER=off) ...' statt der
'#xsn392b PREFETCH-CLAIM-FORM'-Zeilen. Ein Rang, der unter off 'packed'
antwortet, widerspricht der Konstruktion (Registrierung kehrt vor
set_draft_kv_pool UND vor dem Shadow-Marker zurueck, der Cutover entwaffnet)
und waere mit seinen Peers ungleich gepaart -> benannter Gruppen-STOP
(Weg2DraftDisagree 'H51 PREFETCH-CLAIM-FORM' -> _stop_group_from_thread).
Unset/auto/on: die Abstimmung wie bisher, byte-gleich. Kein neuer Schalter.
27B-Linie unberuehrt (anderer Baum); H91d (nicht auf rc2.1l) liest nur
hicache_draft_tier_off und nutzt den Tier nicht.

Tests (test_weg2_hicache_draft_tier_nf_h51.py, Abschnitt H51 rc2.1l, 9):
rot auf d1c7094 (4): Gruppe unter off ohne Form-Kollektiv (TP0+2 Shadows),
eine benannte Zeile, echter prefetch_thread_func-Lauf nur MIN je Operation
bei gleichem Claim 4032, packed-Rang unter off = benannter STOP. Gruen vorher
und nachher (5): on/unset/auto behalten das MAX, Gruppe P ohne off stimmt ab,
ohne Sync-Gruppe keine Kollektive und keine Zeile.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants