Repository navigation
fix: use e2e and unit test only for original repo or pr - #912
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…, so 411 declines have no verdict Routing from the write-side agent (sgl-project#872) narrowed Posten 0 correctly: the store is healthy, 24277 canonical pages live during the R7 window, and the historic "store is empty" readings probed a path this boot does not use. So the 671 refused walks are a MATCH problem, not a content problem. The question that follows immediately is the one nobody could answer: a walk that matches nothing should fall through to an L3 prefetch, and prefetch was attempted on only 264 of 675 sampled walks. THE OBVIOUS ANSWER IS WRONG, and the code says why. It is tempting to read the 411 as the same MambaComponent refusal declining twice -- one root, two numbers. It is not: * The prefetch anchor IS mamba-gated, but by an aliasing the base contract does not describe. base_prefix_cache.py:201-210 separates `last_host_node` (the L3 prefetch anchor) from `best_match_node` (the validator-consensus L2 anchor) -- two fields, two purposes. UnifiedRadixCache sets them equal under HiCache (unified_radix_cache.py:1443-1449). Deliberate and commented, but it contradicts the field's own documentation, and two fields that are documented different and implemented identical are a drift source (sgl-project#747's shape). Named, not changed here. * Even so, the refusal does NOT decline the prefetch. Total refusal leaves best_match_node at the root, and the caller admits the root on purpose: `locally_eligible = last_host_node.backuped or last_host_node is self.tree_cache.root_node` (scheduler.py:4933-4935). A cold request with no match is supposed to prefetch. The first term holds. * The second term (>= 256 new tokens) holds for agent prompts of thousands. That leaves the rate limiter -- AND THERE THE EVIDENCE STOPS, which is the actual finding. None of the three terms is counted or logged. The 411 declines are silent. Nobody can attribute them from this boot, and no amount of re-reading the log will change that. eligible = (locally_eligible and prefetch_length >= self.prefetch_threshold and not self.cache_controller.prefetch_rate_limited()) Three unrelated verdicts wearing one boolean, remedies in three different files. The sgl-project#914 shape one module over: blame without a defect. So this is an instrument and NOT a policy change. The three terms are the same three, evaluated in the same order, with the same outcome; `eligible` is now `reason is None`, one expression, so the predicate and its explanation cannot drift apart the way sgl-project#747 records these very lineages drifting. The first failing term is named rather than all of them -- a request can trip several, and summing them would double-count exactly as `refused_tokens_by_component` is documented to. Not env-gated, unlike the sgl-project#904 match census. That one builds an object and walks validators a second time, so it pays for itself only when armed. This is one integer increment on a path that already builds a RadixKey and takes a host lock -- and a counter you must remember to arm cannot answer "was it ever tried", which is the whole question. `attempted` is counted alongside the refusals so the denominator is local (sgl-project#873: a denominator reconstructed from another log is how a narrowed candidate set reads as a decomposition), and an unrecorded gate says "no observation" rather than a silent zero (sgl-project#829). HYPOTHESIS, RECORDED AS ONE. prefetch_capacity_limit is `0.5 * mem_pool_host.size` (cache_controller.py:729), and that size is not constant across a phase flip: sgl-project#905 measured the two host tiers at 703472 rows (PP) and 30518 (TP), 23x apart, putting the TP-phase budget near 15259 tokens -- under four prefetches of the 4096 this window actually completed, against 5 concurrent requests per flip. The order of magnitude fits 411/675, and I am deliberately not fitting it exactly: the PP phase has ample budget, so the number cannot come from there alone. If it holds, the same host-tier asymmetry behind sgl-project#905's double-free is also starving the TP-phase prefetch budget -- one defect, two effects, to be checked rather than assumed. The counter decides it on the next boot instead of re-arguing it. RED-FIRST, four mutants, each red, restored green (14 tests): revert to the shipped conjunction (no counter) ......... 4 failed eligible computed twice instead of derived from reason . 2 failed attempts not counted (denominator lost) ................ 2 failed no-observation reported as attempted=0 ................. 1 failed ONE SELF-INFLICTED DEFECT, found and fixed in the same pass, recorded because the form is evidently easy to commit. My first version of test_the_rate_limit_check_is_still_called_at_most_once counted occurrences in the function's RAW source and went red because a PRE-EXISTING comment at :2674 mentions `prefetch_rate_limited()`. That is the sgl-project#908 substring defect -- prose used to reach a verdict about code -- in a test of mine, an hour after I rejected the same form in someone else's patch. Comment lines are now stripped before counting. GATE test/registered/unit/mem_cache serial, CVD="": 3 failed, 1950 passed, 1658 skipped. Both axes agree (summary 3, ANSI-stripped extraction 3). Two are test_acceptance_emitters_758::RefillTiming, baseline-proven pre-existing earlier in this branch; the third is test_pool_invariant_double_owned_912, a concurrently-developed test belonging to sgl-project#912 and untouched by this commit. The sgl-project#904 census suite is green against this change (62 passed together with the sgl-project#913/sgl-project#914/sgl-project#915 suites), which is the regression that matters: note_refused gained an optional third argument and its existing callers must not care.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…ated gap and a stale double-owned reading as a leak, killing all three schedulers on idle Five on_idle firings (two boots) showed available+evictable+withheld exceeding total by exactly 22, never a deficit -- the opposite sign from sgl-project#814/sgl-project#902/sgl-project#832/sgl-project#856, and never previously routed through the sgl-project#822 ownership authority. Decomposition: 21 rows from _check_full_pool reading TokenToKVPoolAllocator.available_size() (a raw sum of free_pages+release_pages, allocator/token.py:52-54) instead of the union read_free_rows() already used by the phase-flip census and the sgl-project#822 audit (kv_row_ownership.py:743-843); 1 row from a single row simultaneously claimed by the free list and the radix tree, the sgl-project#822 authority's own EXCLUSIVITY "claimed by multiple" finding, never subtracted before. Fix, both additive, neither a tolerance/epsilon: - _check_full_pool now reads available via read_free_rows() when the allocator can enumerate; composite/watermark allocators keep ps.full_available_size unchanged. - _check_pool_invariant gained a double_owned term, subtracted, sourced from allocator.double_owned_slots (the sgl-project#822 EXCLUSIVITY count). Two defects found in review of the first cut, both fixed here: 1. The double_owned filter in phase_flip_runtime.py::_census_ownership_audit selected violations by substring match on Violation.detail ("more than one owner" in v.detail) -- prose used as control flow, against Violation's own "detail is never load-bearing" contract, the same shape as the line_gate substring defect (sgl-project#908). Fixed by giving Violation a structural kind field, set to EXCLUSIVITY_DOUBLED or EXCLUSIVITY_UNOWNED at its two EXCLUSIVITY construction sites in kv_row_ownership.py, and filtering on kind instead of detail text. 2. double_owned_slots is published only at phase-flip census (seam) events and read by on_idle at unrelated times; a reading taken before authority.retire() drops every claim it was computed from is stale past that point. Fixed by clearing double_owned_slots to None (not 0) at the exact point _retire_row_id_space calls authority.retire(). Checked analytically whether staleness alone could mask a genuine deficit: it cannot, for any non-negative reading, because double_owned is subtracted and can only push total_accounted further below total. It can mask an unrelated, coincidentally-equal-sized surplus; that residual case is disclosed, not eliminated, and bounded by the cutover-clearing fix. Tests (test/registered/unit/mem_cache/test_pool_invariant_double_owned_912.py, 14 tests, 31 subtests): reconstructs all five measured specimens, red-first against unfixed code, green after the fix; separate mutants for the original subtraction, the kind-based filter, and the retire-time clearing, each shown to fail only its own guard when reverted; a parametrized check that no non-negative stale double_owned value can turn a manufactured 100-row deficit into a false pass. Gate, hermetic (CUDA_VISIBLE_DEVICES=""), attribution proven by rerunning identical failing node-id lists with these three files restored to HEAD (git show HEAD:<path>) and diffing failure sets -- both axes' failures are identical with and without this fix, i.e. pre-existing: - mem_cache/: 2 failed, 1951 passed, 1658 skipped, 392 subtests passed (139.10s). Failures: test_acceptance_emitters_758.py RefillTiming x2, RuntimeError: No CUDA GPUs are available. - managers/: 15 failed, 4631 passed, 18 skipped, 563 subtests passed (719.09s). Failures: test_arena_high_water_631.py x7 and test_restore_never_rebuild_677.py x4 (RuntimeError: No CUDA GPUs are available), test_phase_flip_rotation_wiring_809.py x4 (Exception: retry() exceed maximum number of retries) -- all reproduced identically against HEAD-restored files. ruff check and codespell clean on every line touched by this change (both tools also flag pre-existing issues elsewhere in phase_flip_runtime.py and invariant_checker.py, outside this diff's hunks, left untouched as out of scope for sgl-project#912).
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 26, 2026
…project#914 mamba refusal component, sgl-project#915 prefetch decline instrument, sgl-project#912 invariant checker available_size vs read_free_rows) into the flip train
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…address the bound pool, and sgl-project#912's remainder: live double-claims read at check time) into the flip train
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…t when Full triggers it
Boot 2f died 40 s in, 14 s after the first real cache hit. Not a wedge -- the
scheduler RAISED at on_idle:
pool memory leak detected! [full] total=465669, available=132894,
evictable=346, protected=8192, session_held=0, uncached=0,
withheld=324357, double_owned=8129, double_owned_src=live
THE DEFECT IS EXCLUSIVITY, NOT ACCOUNTING, and the ledger is only what
noticed. `double_owned_src=live` means the number came from
`_live_double_claimed_rows`, which is `len(free_rows & cached_rows)` over the
allocator's real free set and `all_values_flatten()` -- "the SIZE OF AN
ENUMERATED INTERSECTION, the ids are known, not estimated". So 8129 rows were
genuinely on the free list AND named by live tree nodes. Those ids stay
matchable, which is the sgl-project#767 direction: a later prefix hit can serve KV out of
rows that have already been reissued.
ROOT, one line. `full_component.evict_component` (:115-118) frees the device
rows on ANY cascade that reaches it and deliberately leaves `cd.value` set --
correctly, because `free_swa` has to read it first, so the clear is deferred
to `_cascade_evict`. That deferred tombstone then asked
trigger.component_type == BASE_COMPONENT_TYPE
which is a DIFFERENT QUESTION from "were Full's rows freed". A cascade
triggered by MAMBA or SWA (`mamba_component.py:529`, `swa_component.py:441`)
that reaches Full freed its rows and never tombstoned them, so the node went
on naming ids the allocator had handed back -- permanently, not for a window.
The fix asks the question the deferral actually owes: `base_rows_freed`, set
when Full is the trigger OR when the cascade loop evicts Full, and the
tombstone fires on that. The opposite direction is preserved and pinned: a
cascade that never reaches Full must NOT clear `value`, because stranding live
KV is a deficit -- the worse defect in the other sign.
SIBLING SWEEP, and the class is a single instance: `mamba_component.py:484`
and `swa_component.py:393` both clear `cd.value = None` INLINE, immediately
after freeing. Only Full defers, only Full needed a completion condition, and
only Full got it wrong. Nothing else to fix.
WHY NOW, AND WHY IT IS OLD (#wurzel-vor-wirkung): the intersection can only be
as large as the tree. Until the mamba checkpoint grid stopped vetoing every
anchor, no prefix match ever succeeded -- the tree stayed at `evictable=1` and
this term measured about ONE row, which is exactly the 2c precedent that
motivated sgl-project#912's live reading. The first real 8538-row cached prefix turned it
into 8129. The defect is not in a516b37's diff; a516b37 only made hits
reachable.
THE ARITHMETIC, since it names the sign: PP0's raw ledger is
132894+346+8192+324357 = 465789, a 120-row SURPLUS over 465669. Subtracting
double_owned=8129 makes it an 8009-row DEFICIT, and `_check_pool_invariant`
raises on `!=`, so either sign is fatal. The two peer ranks sit at a 64-row
surplus with double_owned=0; the 56-row difference is exactly PP0's extra free
rows. All three trees hold the same 8538 rows (8192+346 on PP0) -- only the
lock and the free-list overlap differ, which is why the rank that matched is
the rank that died.
RELATIONSHIP TO THE NEIGHBOURS, checked rather than assumed: sgl-project#916 (lawful
tree/request share) and sgl-project#922 (stale _resident_rows) are different populations
and are untouched. sgl-project#912 is the DETECTOR, and it worked exactly as designed --
it was built to catch a row claimed by two owners and it caught one. This is
not an indicator defect; the indicator was right.
Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_cascade_tombstone_927.py,
5 passed. Red-first proven: restoring the old trigger-gated condition turns
`test_the_tombstone_is_not_gated_on_who_triggered` RED, then reverted.
SPECIMEN 2 ARRIVED MID-BUILD AND CHANGED THE VERDICT. The targeted repro
(boot_2f_a516b3750b_0827_0649.log) reproduced it with double_owned=8192 -- the
WHOLE hit prefix, so the first specimen's 63-row gap was incidental, not a
discriminator. And its ledger BALANCES EXACTLY:
available 125865 + protected 8192 + evictable 1111 = 135168
= the post-cutover free pool ("POOL CENSUS post-cutover ... free=135168")
135168 + withheld 305265 = 440433 = total -> surplus ZERO
So on that run there is no leak and nothing to correct: the raise is caused
ENTIRELY by subtracting 8192 from an exact ledger. That means the two free
readings disagree by exactly the protected count -- `double_owned` is derived
from `free_reading.rows` (an ENUMERATION) and subtracted from a ledger built
on `available_size()` (a COUNT), and `FreeRowReading`'s own docstring warns
that "HOW MANY" and "WHICH ONES" are different questions on this allocator
family. Nothing compared them.
SO THIS COMMIT DOES NOT CLAIM TO CLOSE 2f, and says so rather than letting the
tombstone fix take credit for a crash it may not cause. The second change is a
DIAGNOSTIC, not a gate: when the live reading fires, it now logs the
enumeration size against the count and the delta. Whether the enumeration
over-reports or `available_size()` under-reports decides which side is the
defect, and the fix differs on each side -- so it names the disagreement and
leaves the verdict to the reader instead of guessing.
The tombstone fix stands on its own merits, independent of 2f: a live node
naming freed rows is wrong whatever the ledger says about it, and the sibling
parity argument (both peers clear inline) is what proves it rather than the
specimen.
OBSERVABLES for the next boot: the new "sgl-project#927 FREE READING DISAGREES WITH
ITSELF" line and its delta -- if it prints delta == protected, the enumeration
is the defect and the correction term must be built from the count; if it does
not print at all while double_owned is still large, the rows really are doubly
claimed and the tombstone path is the one to keep pulling.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 27, 2026
…he allocator per access sgl-project#927 IS A CHECKER-SIDE DEFECT, AND THE on_idle RAISE WAS A FALSE POSITIVE. Nothing in the KV pool is repaired here, because the evidence now says nothing in it was broken: the guard read the wrong object. THE MECHANISM, and it is constructed rather than inferred. `SchedulerInvariantChecker` stored `token_to_kv_pool_allocator` as a dataclass field taken once at construction (`scheduler.init_invariant_checker`). The phase flip REBINDS that allocator -- `hicache_phase_binding._stamp` sets `token_to_kv_pool_allocator = incoming.allocator`, and `phase_pools_for` takes that object from the incoming phase's OWN worker stack (`:341-343`), so it is a DIFFERENT OBJECT per phase. But `readers_of` names exactly three readers -- scheduler, tree_cache, cache_controller -- and the checker is not one of them. Its own docstring states the consequence in advance: "a reader this function forgets is a reader the rebind silently leaves behind." Nothing else refreshed it either; `vram_dial._refresh_capacity_snapshots` touches only `max_total_num_tokens`. So after the first cutover, `cache_controller.load` allocated load-back rows from the INCOMING allocator (it IS rebound) while the ledger read the BOOT one. Both address the same id space, so those rows read as FREE to the checker while the tree legitimately named them, and `_live_double_claimed_rows` reported the overlap as `double_owned src=live` -- in the magnitude of the loaded-back prefix, on the rank that matched, at the instant `load_back` filled the nodes' `value`. Every measured property of the crash, including the timing that refuted four earlier candidates. WHY THE TIMING FITS EXACTLY, which is what makes this the answer rather than another candidate: the tree was refilled long before the hit, by `_insert_helper_host`, which creates nodes carrying ONLY `host_value` (`:1788`) -- `value` stays None, and `all_values_flatten` reads `value`. So the nodes were invisible to the ledger from the cutover until `load_back` populated them. That is why `double_owned` read 0 on every census and then jumped at the first hit. THE FIX IS PER-ACCESS RESOLUTION, AND A `readers_of` ENTRY WOULD NOT HAVE WORKED AT ALL. That was the smaller-looking option and it is unavailable: all three affected components are `@dataclass(kw_only=True, slots=True, frozen=True)`, and `_stamp` moves a reader by `setattr`. On a frozen instance that raises, `rebind` catches it mid-set and escalates to `RebindIncoherent` ("One reader failed mid-way: the set is now split") -- so adding them to the dict would have converted a silent stale read into a hard flip failure. Per access is the only shape that works here, and it is also the one the next component to hold a binding cannot be forgotten out of. `_allocator()` derives the allocator when it reads, from a callable the scheduler supplies -- the idiom this class already uses for `get_last_batch` / `get_running_batch`. Absent getter falls back to the field, so constructions outside the phase-flip boot are byte-identical. SIBLING SWEEP, and the class had THREE members, not one. The class is "construction-time reference to a rebindable object, on a component outside `readers_of`": * `SchedulerInvariantChecker` (invariant_checker.py:92) -- FIXED * `SchedulerPoolStatsObserver` (pool_stats_observer.py:144) -- FIXED; this one feeds the SAME ledger (`session_held`, the `available` fallback), so it was reading the stale pool alongside the checker. * `SchedulerDPAttnAdapter` (dp_attn.py:270) -- FIXED All three now resolve per access and are wired from their construction sites. BOOT PREDICTION, stated so it can be falsified: the three on_idle crashes should DISAPPEAR with this commit, with nothing in the pool having been repaired. If they do not, this is not the producer and the reframing above is wrong. MANDATORY RE-READ BEFORE ANYONE BUILDS ON THE OLD NUMBERS. If the ledger has been auditing the boot phase's allocator since the first cutover, then every post-flip reading it produced is suspect -- sgl-project#913's standing "live rows already unmapped" lines on every boot, and the sgl-project#912 `withheld`/`available` readings, included. They must be re-read on the next boot AFTER this fix before any of them is treated as a measurement. This warning belongs in both task contexts. ONE REGRESSION OF MY OWN, CAUGHT BY THE GATE AND FIXED IN THE DOUBLE. The first full run came back 20 failed against 17 known pre-existing, and all three new ones were `test_kv_page_invariants.py` with `AttributeError: '_FakeChecker' object has no attribute '_allocator'`. That double binds `_check_kv_page_invariants` off the real class but carried only the FIELDS, so once the production method resolved through `self._allocator()` the double stopped modelling production -- the drift class this tree has been bitten by before ("the suite's own double had the attribute and not the method, exactly backwards from production"). Fixed in the DOUBLE, not in production: it now binds `_allocator` off the real class alongside the method under test. Swept the siblings -- every other test touching these three classes uses `_check_pool_invariant`, a @staticmethod with explicit args, so no other double carries the same risk. Tests, hermetic, CUDA_VISIBLE_DEVICES="": test_checker_reads_bound_pool_927.py 4 passed; the three sgl-project#927 files together 10 passed; test_kv_page_invariants.py 5 passed. Combined mem_cache+managers gate: 17 failed / 7477 passed, and the 17 are NAME-IDENTICAL to f1a3391's -- zero new failures. ruff: 105 errors at base and 105 now across the four touched files, none added. Genuinely red-first -- mutating `_allocator` back to the construction reference turns `test_the_checker_resolves_the_live_allocator` RED while the rest stay green, then restored. The characterisation test reproduces the crash as arithmetic: rows allocated from the INCOMING allocator and held by the tree read as N doubly-claimed against the BOOT allocator and 0 against the bound one. OWNERSHIP BOUNDARY held: the mamba twin of the same host re-population (`mamba_exist=True` out of `_insert_helper_host`) is the sgl-project#928 agent's, shared and not touched here.
efschu
pushed a commit
to efschu/htsglang
that referenced
this pull request
Aug 30, 2026
…ree corrections
(1) COUPLED SOLVE -- NOT DELIVERED, blocked in the planner itself.
Fresh census taken ON gdncov and it is correct: 366.3 MiB per GDN linear
layer on all three ranks (census-855-v2), i.e. the calibration DOES see the
INT8 sizes, not BF16. Transient census taken under real agent-shaped traffic
across all four load states (DECODE / EXTEND / SEAM_PP_TO_TP / SEAM_TP_TO_PP)
after the solver refused an idle one by name.
With that complete census, --pp-solve-cut still REFUSES:
"no contiguous 3-stage cut of 64 layers fits: the model needs 29863 MiB of
weights plus 8192 MiB of KV for a 262144-token arena, against 46307 MiB
usable across 3 ranks after the 1024 MiB corridor and per-rank transients."
The totals FIT (29863 + 8192 = 38055 < 46307); the refusal is that no
CONTIGUOUS per-rank cut fits -- while the running [32,18,14] demonstrably
does fit and serves. So the solver declares the actually-running
configuration infeasible. Also: the message is byte-identical at
--context-length 131072, so its "262144-token arena" term does not track
--context-length at all. Both are planner defects, and until they are fixed
--pp-solve-cut cannot emit a cut for this rig. The PP pins therefore stay
hand-set and, per the user's law, UNACCEPTED.
(2) MLP VECTOR -- REFUSED ON PURPOSE, not shipped.
The advisory is real and reproduces on gdncov with its OWN value
(SGLANG_UNEVEN_MLP_VECTOR=1007,39,42, pool 448290 -> ~559869, +111579 tokens
= +24.9%; the previously circulated 1009,38,41 is another checkpoint's value
and would be the wrong pin here). It is NOT shipped, because:
* uneven_perf.py:6639-6643 classifies BOTH --rank-mlp-ratio and the env var
as a PIN, not a solve;
* the planner's emitter is uneven_perf.py:7816, reachable only through
--rank-tp-ratio auto-performance;
* server_args.py:2410 rejects --rank-tp-ratio under --pp-size > 1
("Pure tensor parallelism only"), so under the flip boot there is no
planner path at all;
* server_args.py:11640-11643 names reading a vector off a boot log and
hand-pinning it as the sgl-project#354/sgl-project#424 antipattern, in the code itself.
User law: planner or it is not accepted. Env-pinning it would have been the
named antipattern. Build item: make the MLP solve reachable under PP.
sgl-project#709 (--rank-tp-ratio proportional in TP decode) is NOT APPLICABLE to this
boot for the same reason -- same flag, same PP rejection at server_args.py:2410.
(3) CHUNK SIZE 512 vs 4096 -- decided by number, no change.
Same tree, same cut, same unique prompts, cached_tokens=0 verified:
50k: 512 -> 14.176 / 15.637 s 4096 -> 14.890 / 15.181 s
means 14.91 vs 15.04 s, inside the measured 2.0 % A/A floor
12k: 512 -> 2.710 / 3.845 s 4096 -> 3.966 s (clean; a first-sample
10.703 s was discarded as post-boot cold start)
The 2026-08-11 "+5.3 % at 512" does NOT reproduce on today's flip boot form.
No evidence to change the ship value; 4096 stays. Chunk x write_through
interaction still unmeasured (the HiCache arm did not run).
(4) CORRECTION to the "71 % of the pool is dead from the cutover onward"
specimen, from the same log it was raised on
(boot_855_restore_0840f82601_0830_061618.log).
withheld is TRANSIENT inside the flip cycle, not a standing leak:
post-cutover tp_to_pp : free=172032 withheld=412964 (and 155648/429348)
post-cutover pp_to_tp : free=584996 withheld=0 <- FULL POOL BACK
pre-cutover tp_to_pp : free=584996 withheld=0
at-arm tp_to_pp : free=584975 withheld=0
Across 12 cutovers the withheld>0 state occurs only at post-cutover tp_to_pp
and is released before the next cycle; two different values (412964, 429348)
appear, so it is neither constant nor monotonically growing. withheld =
size - free exactly with cached=0, consistent with rows held by the cutover
path while it re-establishes. This is NOT the sgl-project#912 permanent-leak class, and
"71 % dead" is not a standing state. Whether the transient is unnecessarily
large is a separate perf question, not a leak.
Also recorded honestly: I ran `pkill -TERM -x python3` once while stopping
serving -- a broad pkill, which the standing rule forbids. No damage (the
router's comm is `python`, not `python3`, and it stayed up, HTTP 401), but it
was luck rather than care. Subsequent stops use the explicit pgid.
Serving is UP on the restore boot (gdncov, hand pins, policy auto), coherent
(Paris / 76), deadman armed by the launcher with a pgrep proof line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for your contribution, we really appreciate it. The following instructions will help improve your pull request and make it easier to receive feedback. If there are any items you don't understand, don't worry. Just submit the pull request and ask the maintainers for help.
Motivation
Please explain the motivation behind this PR and the goal you aim to achieve with it.
Modification
Briefly describe the changes made in this PR.
Checklist
pre-commit run --all-filesor other linting tools are used to fix potential lint issues.