Repository navigation
Add Gemma2 - #592
Merged
Merged
Add Gemma2#592
Conversation
merrymercy
requested changes
Jul 5, 2024
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
Co-authored-by: Lianmin Zheng <lianminzheng@gmail.com>
merrymercy
reviewed
Jul 5, 2024
| last_logits /= self.config.final_logit_softcapping | ||
| last_logits = torch.tanh(last_logits) | ||
| last_logits = last_logits * self.config.final_logit_softcapping | ||
| last_logits *= self.config.final_logit_softcapping |
Collaborator
There was a problem hiding this comment.
last_logits.mul_(self.config.final_logit_softcapping)
|
|
||
| if hasattr(self.config, "final_logit_softcapping"): | ||
| last_logits = last_logits / self.config.final_logit_softcapping | ||
| last_logits /= self.config.final_logit_softcapping |
Collaborator
There was a problem hiding this comment.
last_logits.div_(self.config.final_logit_softcapping)
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
…ject#592) * Copy OSS code from 934ae89 on 20260112 * update pr-test.yml * update * update test framework * update * update * update * update * update * lower threshold to unblock --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Wei Gong <weigong@sillysnake.cloud.together.ai> Co-authored-by: gongwei-130 <weigong@research-dev-b200-08.cloud.together.ai>
iridiumine
pushed a commit
to chenxu214/ci_sglang
that referenced
this pull request
Jul 27, 2026
Wire torch.ops.npu.causal_conv1d (sgl-kernel-npu sgl-project#592) into the KDA backend's NPU path as an opt-in alternative to the legacy causal_conv1d, gated by env var SGLANG_KDA_ASCENDC_CONV1D (default "0" = legacy; "1" = new). - Module-level switch _USE_ASCENDC_CONV1D; import sgl_kernel_npu in the is_npu() block as a robust libsgl_kernel_npu.so load trigger. - forward_decode (run_mode=1) and forward_extend run_conv (run_mode=0, 3-call q/k/v): call the new op under the switch; legacy path kept verbatim as the else branch (decode currently uses torch_causal_conv1d_update_npu). - _get_conv_weights_for_op: per-layer cached [width,dim] BF16 conv weights. K3 checkpoints store conv weights FP32; the Ascend-C op requires BF16, so this downcast is the precision change (accuracy-validated on NPU). - conv_states fed to the op as [pool, state_len, dim] (transpose + writeback); decode x made contiguous for the op's TORCH_CHECK. No behavior change with the switch off (default).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 5, 2026
…tion gap, gate stays with assertions
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 9, 2026
…e read/write seam Groundwork for making the FULL KV budget available to whichever phase is active, instead of splitting it between two permanently resident pools. MHATokenToKVPool gains swappable_backing: allocate on a VA reservation (the existing KvVmmBufferOwner) and back the whole span immediately, then release_backing()/restore_backing() unmap and remap the physical pages behind unchanged addresses. Sizing is untouched -- self.size stays what the constructor computed, already DCP-translated by _dcp_token_sharded_pool_rows, so the sgl-project#592 resize-translation gap on finalize_backing does not apply: the span released and restored is always the pool's own row count. That is deliberately NOT post-capture sizing, which is gated off for this config anyway (dcp_size == 1 and enabled prefill graphs, neither true here). PhaseFlipRuntime gains pre_write_fns, fired at the read/write seam. That seam is the only instant where backing may move: every byte the transition owes has been read (peer legs materialised, local leg hoisted in 93ee1e2310) and the destination has not been touched. Empty by default, so this commit changes no behaviour. Tests: scripts/run_631_flip_family.sh 271/271 (+1 seam-ordering pin, per rank because the ranks run concurrently and a global event order would interleave and prove nothing). Not yet wired: the pools are not constructed with swappable_backing and nothing registers a swap -- arming it needs the tree-cache/allocator binding question settled first (they are bound to the PP stack's pools for process life), which is the next step.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…lden statt bauen (Koordinator-Klausel) Koordinator-Entscheid: post-capture sizing auf den Budget-Pfad bringen, mit der Klausel "Wenn der Fix mehr als S/M wird: melden statt bauen". Gescoped; die Klausel greift. Kein Code geschrieben. WARUM DER ENV-SCHALTER ALLEIN NICHT REICHT. `SGLANG_ENABLE_POST_CAPTURE_KV_SIZING` ist EnvBool(False) (environ.py:489) und in keiner Boot-Env gesetzt -- das ist der erste Verweigerer. Der zweite ist `dcp_size == 1` in `post_capture_kv_sizing_planned` (server_args.py:14345), und der bindet genau dort, wo es weh tut. Gemessen an der Boot-Reihenfolge (boot_855_lmhead5_..._093325.log): 09:33:55 Sizing-Pass 1, sgl-project#797 sagt dcp_size=1 (PP-Layout) 09:34:07 cuda_graph_capture: 0.000-0.001 s <- de facto NO-OP 09:35:53 Sizing-Pass 2, sgl-project#797 sagt dcp_size=3 (TP/Flip-Layout) 09:36:10 cuda_graph_capture: 6.34-6.39 s <- DIE echte Capture 09:36:11 OOM, 810 MiB Die 810-MiB-Capture liegt VOLLSTAENDIG im dcp_size=3-Regime. Der env-Schalter wuerde also nur den Pass freischalten, dessen Capture 1 ms kostet, und den teuren Pass weiter verweigern lassen. WAS OPTION 2 DAMIT WIRKLICH VERLANGT -- die sgl-project#592-DCP-Sperre heben. Der Kommentar an server_args.py:14296-14344 benennt die fehlende Arbeit selbst und praezise: `pool.finalize_backing()` reicht `config.max_total_num_tokens` direkt als Zeilenzahl an `_finalize_backing_tokens` weiter, waehrend alle fuenf Pool-Konstruktionsstellen ueber `_dcp_token_sharded_pool_rows` gehen; die Identitaet gilt nur bei dcp_size == 1. Ein Heben OHNE diese Uebersetzung "would back C rows per rank instead of the rank's compact rows and set pool.size to a number the owner rule never addresses" -- also STILL falsche Groesse, kein Crash. Auf der Achse uneven-DCP-weighted, die laut Nutzergesetz nie abgeschaltet wird. ZWEITER BEFUND, der die Kostenschaetzung des Kommentars schwaecht: derselbe Kommentar beruft sich auf `test_post_capture_dcp_gap_592`, der "beides pinnt, die Sperre und die fehlende Uebersetzung, damit wer es angeht von der Evidenz aus startet". DIESER TEST EXISTIERT NICHT. `git grep` ueber den ganzen Baum: EIN Treffer, und das ist der Kommentar selbst. Das Sicherheitsnetz, auf das "bounded work" sich stuetzt, ist nicht da. (Klasse: PRAESENT-ABER-UNVERDRAHTET vs ABWESEND -- hier schlicht abwesend.) DRITTER BEFUND, aus dem sgl-project#797-Kommentar (mixin:7236-7248): auf einem Phase-Flip-Boot laeuft dieses Sizing VOR `build_phase_flip_tp_stack`, der Token-Vektor ist noch nicht installiert -- und "the site that DOES have the measured capacities (post_capture_resize_kv_pool) is HINT-ONLY". Die Stelle, die der Entscheid autoritativ machen will, ist auf der Flip-Form ausdruecklich nur Hinweisgeber. Sie autoritativ zu machen ist ein dritter Arbeitsblock, nicht Teil der Zeilen-Uebersetzung. SUMME: env-Default + sgl-project#592-Zeilen-Uebersetzung + Hint-only->autoritativ auf der Flip-Form, ohne den Test, den der Kommentar als vorhanden annimmt, auf einer Achse mit stiller Fehlgroesse als Fehlermodus. Das ist L, nicht S/M. ZUR BEGRUENDUNG (a) DES ENTSCHEIDS, unveraendert richtig: ein portierter slack_gb-Fraction-Term auf dem MiB-Pfad waere die Schaetz-Form, die der Pfad abschaffen sollte. Der Einwand gegen Option 1 bleibt gueltig -- er macht Option 2 nur nicht kleiner. VORSCHLAG ZUR ENTSCHEIDUNG (nicht gebaut): den 810-MiB-Posten als PROVISORISCHE Schranke auf dem Budget-Pfad buchen, ausdruecklich als Uebergang kommentiert und mit dem gemessenen Wert statt einer Fraction -- das ist S und macht das lm_head-Artefakt sofort bootbar -- und die sgl-project#592-Hebung als eigenen Posten fuehren, beginnend mit dem fehlenden Test. Entscheid liegt beim Operator. BELEG-STUFE: DESK-BEWIESEN (Praedikat-Zeilen, git grep) + BOOT-GEMESSEN (Capture-Dauer je Pass, sgl-project#797-dcp_size, beides aus den Logs dieses Strangs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Logits aligned with Huggingface when using float16. Our default is bfloat16.