Skip to content

docs: add llama 3.1 405b instruction - #739

Merged
zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:405
Jul 26, 2024
Merged

zhyncs merged 1 commit into
sgl-project:mainfrom
zhyncs:405

Conversation

@zhyncs

@zhyncs zhyncs commented Jul 26, 2024

Copy link
Copy Markdown
Contributor

Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.

Motivation

Please explain the motivation behind this PR and the goal you aim to achieve with it.

Modification

Briefly describe the changes made in this PR.

Checklist

  1. Ensure pre-commit pre-commit run --all-files or other linting tools are used to fix potential lint issues.
  2. Confirm that modifications are covered by complete unit tests. If not, please add more unit tests for correctness.
  3. Modify documentation as needed, such as docstrings or example tutorials.

Co-authored-by: Ying1123 <sqy1415@gmail.com>
@zhyncs
zhyncs merged commit 05d216d into sgl-project:main Jul 26, 2024
@zhyncs
zhyncs deleted the 405 branch July 26, 2024 11:03
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Co-authored-by: Ying1123 <sqy1415@gmail.com>
cen121212 pushed a commit to cen121212/sglang that referenced this pull request Nov 10, 2025
<!-- Thank you for your contribution! Please follow these guidelines to
enhance your pull request. If anything is unclear, submit your PR and
reach out to maintainers for assistance. Join our Slack community at
https://slack.sglang.ai to discuss further. -->

## Motivation

<!-- Describe the purpose and goals of this pull request. -->

## Modifications

<!-- Detail the changes made in this pull request. -->

## Accuracy Tests

<!-- If this pull request affects model outputs (e.g., changes to the
kernel or model forward code), provide accuracy test results. -->

## Benchmarking and Profiling

<!-- If this pull request impacts inference speed, provide benchmarking
and profiling results. -->

## Checklist

- [x] Format your code according to the [Format code with
pre-commit](https://docs.sglang.ai/developer_guide/contribution_guide.html#format-code-with-pre-commit).
- [x] Add unit tests according to the [Run and add unit
tests](https://docs.sglang.ai/developer_guide/contribution_guide.html#run-and-add-unit-tests).
- [x] Update documentation according to [Write
documentations](https://docs.sglang.ai/developer_guide/contribution_guide.html#write-documentations).
- [x] Provide accuracy and speed benchmark results according to [Test
the
accuracy](https://docs.sglang.ai/developer_guide/contribution_guide.html#test-the-accuracy)
and [Benchmark the
speed](https://docs.sglang.ai/developer_guide/contribution_guide.html#benchmark-the-speed).
cherryblo added a commit to cherryblo/sglang-project that referenced this pull request Jul 2, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…- unstrangle the funder

My sgl-project#744 fix has a live regression and this is the refinement. Specimen
21:46:32 on 72696b0aec: tp_to_pp refused 35x, IDLE-LOCK with 407,622 tokens
pending and 0 resident, the guard reporting "no KV provider is registered"
because the armed-only gate refused the rung at BOTH sites and left the
provider list empty. Seam staging (PP1: needs 1269 MiB, spendable 883, rung
wanted SHRINK to 126506 with slack 45526 = a 1455 MiB deficit that WAS
coverable) could never be funded.

THE ERROR WAS MINE AND IT WAS A DESIGN ERROR, not a slip. Seam-funding
eviction of recomputable prefix rows is REQUESTED BY the flip machinery -- it
is what "KV capacity is the funder" means -- so refusing the rung for the
duration of a flip protects against the 21:18 crash by disabling the thing the
flip is waiting on. A wholesale gate cannot be right when the protected party
and the requesting party are the same machinery.

THERE WERE TWO STRANGLE POINTS, not one. The armed gate was the obvious half.
The second was quieter: _nothing_resident() returning False on a parked extent
dropped through to the unknown-refuse branch, so even with the gate removed the
rung would still have declined. Both are fixed.

THE REFINEMENT. The parked extent already carries exactly the information
needed to be selective, which is why this is clean rather than a compromise:
rows INSIDE it are what the flip is about to pack and may not be touched; every
row ABOVE it is recomputable prefix and is precisely what the funding wants. So
the extent PINS THE CEILING instead of closing the rung. _parked_ceiling()
returns the highest parked row id, -1 when nothing is parked, and -2 for the
one case that still refuses wholesale: an UNKNOWN extent while a flip is armed,
where there is no boundary to name. Both call sites take req_max =
max(req_max, parked), so the evictor is handed a ceiling that cannot reach into
the extent.

UNKNOWN, DECIDED AND DOCUMENTED as asked: it refuses ONLY while a flip is
armed. Outside a flip there is nothing parked to protect, and closing the rung
there is exactly the strangle this commit removes. sgl-project#746 (the exact arm-time
snapshot) is what removes this last wholesale case; until then it is one
narrow, named condition rather than the whole flip window.

TEST RESULTS

test_evict_rung_flip_park_744.py refined to the new semantics, 19 tests + 2
subtests green. The two tests that asserted the wholesale gate are REPLACED
rather than deleted -- they now assert the funding path delivers -- and the
crash protection gets its own explicit test that the evictor's ceiling never
reaches into the parked extent.

Mutation matrix, 4 mutants, ALL KILLED, covering both directions:
  N1 exclusion removed (parked ignored)      -> 1 failed  CRASH protection dies
  N2 exclusion covers everything (gate back) -> 1 failed  FUNDING dies
  N3 collecting site ignores parked ceiling  -> 2 failed
  N4 unknown-while-armed no longer refuses   -> 3 failed
N1 and N2 are the (c) pair: neither over- nor under-covering survives.

Pinned suites green together, 142 passed + 87 subtests: sgl-project#744, sgl-project#731 + the
#731x#744 interaction pin, both sgl-project#717 suites, sgl-project#656, sgl-project#684, sgl-project#662-f4, sgl-project#713
admission intake, sgl-project#739 prefill progress.

Ruff and codespell clean.

PP2 SECONDARY, checked and NOT claimed as fixed. The 0 MiB shrink against an
unmoved driver-free column is an already-instrumented condition
(kv_backing_relief.py:1915-1927): the code detects it, names retained handles
(SGLANG_FLIP_SEAM_RETAIN_HANDLES) or a missing commit chunk as the cause, keeps
the cap on and returns 0. The gate refinement lets the rung PRICE again; it
cannot make a retained-handle arena PAY. Necessary, not sufficient -- the
retain setting needs verifying at the boot, and this commit does not pretend to
close it.

No boot.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 17, 2026
…ip named

Pass 2's state was unverified after the process-exit waves, so this pass began
by CHECKING every worklist item against the tip rather than by merging. Five of
eight were already on the train, verified with merge-base --is-ancestor rather
than assumed:

  feat/407-registry-reconcile (ahead=0) + docs/407 + VERDICT_407 present
  the two DESIGN_407 corrections filed under sgl-project#732 -- both committed
  fix/485-gdn-family-report (ahead=0)
  feat/363-remainder (ahead=0)  <- the worklist flagged this MISSING. STALE.
  fix/602-fill-side (ahead=0)
  the sgl-project#710 adapter item -- the named site carries the repair, from sgl-project#540

Merged (all clean, zero conflicts):
  origin/train/0817-desk  -- sgl-project#737 ack drain, sgl-project#734 dead-peer discriminator,
                             sgl-project#721 help text, and sgl-project#738 direct-io which F4-r4
                             pushed during the pass, resolving that item
  feat/516-miss-slot-budget
  feat/553-remainder
  fix/739-prefill-progress-signal

Plus sgl-project#748 shipped ON the train before the freeze: my sgl-project#744 armed-only gate
strangled the flip's own funder (35 refused tp_to_pp, IDLE-LOCK at 407,622
tokens pending), and the review boot must not carry the unrefined gate.

TEST EVIDENCE, per-suite and compared by FAILED/ERROR line rather than total:
  baseline before any merge        57 failed, 4 errors, 5510 passed
  after the three main merges      57 failed, 4 errors, 5575 passed
  new failures ZERO, disappeared ZERO
Pinned suites after sgl-project#748: 142 passed + 87 subtests.

ONE FINDING RECORDED RATHER THAN RE-RUN AWAY. Three later full-scope runs
showed ~50 extra failures in the distributed/ collective-floor family, and the
obvious suspect was the last merge. sgl-project#739 is EXONERATED by a same-commit
contradiction: commit 3c49cbc, the tree BEFORE sgl-project#739, was run twice on the
identical scope -- once ZERO new, once FIFTY. Same commit, same command,
opposite results. Corroborated: the suites pass in isolation, pairing each new
suite with a victim stays green, and distributed/ alone gives 27 rather than
27+50. So the combined run is order-dependent at ~50 tests on this lineage:
a test-infrastructure defect worth its own ticket, not a property of this
train. A suite that gives fifty different answers on one commit cannot gate
anything, and the tempting move -- re-running until it looked green -- would
have buried that.

NOT on the train: Slot-2's sgl-project#747, which arrived after the freeze. Nothing else
was withheld; every worklist item is merged or verified present.
efschu pushed a commit to efschu/htsglang that referenced this pull request Oct 23, 2026
…ueue clock (Qwen review gap of ee5fb03)

A 25 s old queue with a prefill chunk 1 s ago does not alarm; the same
timeline without the prefill-progress signal still alarms. Written by a
Qwen agent (deskq/done/qwen-test-flipverdict-739-1023.py), taken over
unchanged. pytest_gedeckelt: 2 passed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant