Skip to content

Gemini Backend - #9

Merged
Ying1123 merged 10 commits into
mainfrom
gemini
Jan 17, 2024
Merged

Ying1123 merged 10 commits into
mainfrom
gemini

Conversation

@caoshiyi

Copy link
Copy Markdown
Contributor

add support for gemini backend

@caoshiyi
caoshiyi requested a review from Ying1123 January 16, 2024 00:33
@merrymercy
merrymercy force-pushed the main branch 4 times, most recently from 5a84411 to 97b04d4 Compare January 16, 2024 05:44
Comment thread python/sglang/lang/ir.py Resolved
Comment thread python/sglang/backend/gemini.py Resolved
Comment thread examples/quick_start/images/cat.jpeg Resolved
Comment thread examples/quick_start/images/rat.jpeg Resolved
Comment thread python/sglang/lang/interpreter.py Resolved
@Ying1123

Ying1123 commented Jan 17, 2024 •

Copy link
Copy Markdown
Contributor

@Ying1123
Ying1123 merged commit fd7c479 into main Jan 17, 2024
@Ying1123
Ying1123 deleted the gemini branch January 17, 2024 06:29
Ying1123 pushed a commit that referenced this pull request Sep 13, 2024
@0xymoro 0xymoro mentioned this pull request Nov 10, 2025
5 tasks done
yhyang201 pushed a commit that referenced this pull request Dec 13, 2025
* test: add EPD disaggregation integration tests

* fix comment for encoder-only

* revert http_server warmup for vlm
fstandhartinger pushed a commit to fstandhartinger/sglang that referenced this pull request Jan 13, 2026
tpoisonooo pushed a commit to tpoisonooo/sglang that referenced this pull request Feb 12, 2026
chx96642264 pushed a commit to chx96642264/sglang that referenced this pull request Mar 11, 2026
rucnyz added a commit to rucnyz/sglang that referenced this pull request Apr 30, 2026
…s 28 xfers

v9 pool-binding-shift trace produces real differentiation:
- Phase B (KV-bound 8K random): L1+L2 -37% mean TTFT vs stock
- Phase C (mixed 4K random):     L1+L2 -38% median E2E vs stock
- Cross-pool transfers: stock 0, L1-only 0, L2-only 0, L1+L2 28

Two surprising findings documented:
1. Layer 2 alone fires zero transfers — Layer 1 retention is what
   makes Layer 2 cross the firing threshold.
2. Phase A regresses with L1 (-20% TPS) because K_big=8192 hurts on
   prefix-friendly GSP. Consistent with A2's K_big=0-wins finding.
   Adaptive K_big control marked as follow-up.

Settings status: Setting 1 marked **DONE v6 NULL + v9 PASS**.
All 4 user-requested follow-ups (sgl-project#9 Q3.A 4-arm, sgl-project#10 Sweep 1
multi-seed, sgl-project#11 Setting 4 fallback rule, sgl-project#12 Setting 1 v9 trace)
now complete.
predator2k pushed a commit to predator2k/sglang that referenced this pull request May 12, 2026
Second independent review of the TTExecutionBackend amendment caught
issues the first round missed. All addressed here.

CRITICAL:
- test_env_var_selects_backend called `.get.cache_clear()` on
  envs.SGLANG_TT_EXECUTION_BACKEND.get, but EnvField.get (environ.py:54)
  is a plain method that reads os.environ on every call — no lru_cache,
  so the call AttributeError'd. Verified via direct read of environ.py.
  Removed the spurious cache_clear() call; updated the test docstring
  to record the verification.
- Same test referenced `envs.X.get()` but the test file's import block
  didn't `from sglang.srt.environ import envs`. Side-effect of the
  cache_clear removal: the only `envs` reference is now gone, so no
  new import line needed — the dependency vanishes with the fix.

IMPORTANT:
- Spec §5.1 worker pseudocode still used `self.wrapper.X` / called
  `ttnn.to_torch(logits_ttnn)` in the worker. Plan G.2 was already
  updated to `self.execution_backend` returning host tensors directly.
  Synchronized spec §5.1 to match: worker calls
  `self.execution_backend.{extend,decode_step}`, gets host torch
  tensors, never imports ttnn (invariant sgl-project#7 enforcement).
- Spec §2 prereq (line 85) still mentioned "Phase F's TTLlamaWrapper
  glue" as the target of the Phase 0.2 API capture. Updated to
  TTTransformersExecutionBackend with pointer to the amendment note.
- Spec §12 open question sgl-project#9 still referenced TTLlamaWrapper. Updated;
  noted that the ABC + registry remain valid even if the
  tt_transformers adapter has to be redone.

NIT:
- noqa: E402 comment on the registry __init__.py snippet is now
  inline with a leading explanatory comment block; future maintainers
  shouldn't accidentally silence E402 elsewhere.
- Plan F.1 Step 6 says `git rm`, not plain `rm` — preserves blame
  traceability from tt_transformers_backend.py back to commit
  152b67c (the original llama_adapter.py).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
yushengsu-thu referenced this pull request in yushengsu-thu/sglang May 19, 2026
BCG was the fallback path designated for cases where PCG hits a
torch.compile / Inductor incompatibility (B5/B6 in the design doc).
Adds LoRA-aware ForwardBatch construction and prepare_lora_batch hooks
to mirror what we did for PCG, so --enable-breakable-cuda-graph also
works under --enable-lora.

Changes (breakable_cuda_graph_runner.py):

* _build_capture_forward_batch: stamp lora_ids=[None]*bs when
  --enable-lora is set (was hard-coded None). Same semantics as PCG:
  empty ids drive the kernels to launch but early-return on rank=0.

* New _maybe_prepare_lora helper that, when enable_lora and the batch
  has lora_ids, calls lora_manager.prepare_lora_batch(forward_batch,
  force_cuda_graph=True). This populates the pinned cuda_graph_batch_info
  that gets baked into the captured BreakableCUDAGraph.

* _warmup and _capture_one now call _maybe_prepare_lora before
  init_forward_metadata, mirroring PCG's _capture_one path.

* replay() re-runs prepare_lora_batch(force_cuda_graph=True) at the top
  so live batch's adapter metadata lands in the same pinned buffer
  (init_new's earlier non-cuda-graph alloc is overwritten).

BCG bypasses torch.compile entirely (it uses raw torch.cuda.CUDAGraph
plus manual @eager_on_graph break points), so it should not hit the
Dynamo-vs-Triton-config issues that block chunked + PCG and MoE + PCG.

End-to-end smoke for BCG path still TODO (would require
--enable-breakable-cuda-graph + --enforce-piecewise-cuda-graph). The
PCG path is the production default and is already validated by smoke
#9 (Qwen3-8B + Qwen3-8B all-linear adapter + triton backend).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
szewang pushed a commit to szewang/sglang that referenced this pull request May 26, 2026
@harmya harmya mentioned this pull request Jun 25, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants