Repository navigation
Key a backup's wrong-cluster alert on its endpoint entry (finding 130) - #147
Merged
Merged
Conversation
shuv-amp
changed the base branch from
auto/A/sweep-copy-every-account
to
main
October 19, 2026 17:55
Eight Vault checks guard reads of program-owned records, and no test could fail when any one of them was deleted: - price_table (custody.rs:224-227): an asset record prices a release only if it names this protocol (F1) and the mint asked for (F2), and sits at its own PDA (F3). - load_protocol (lib.rs:306): the protocol record at its own PDA (F4). - load_vault (lib.rs:323): the vault record at its own PDA (F5). - read_backing (custody.rs:64-66): an owner's backing record is read only if it names this protocol (B1) and that owner (B2), and sits at its own PDA (B3). No instruction writes such a record. The Vault creates each one at its PDA from checked values, with distinct discriminators. So these are defence in depth, and the tests write forged accounts with set_account. That is the pattern already used for the re-owned vault token account and the planted protocol account. Each forged record fails one term and passes the others. Under F1-F3 and F5, a quote with the forged record succeeds and promises 1,000 USDC backed. Under F4, the copied protocol is still refused, but by load_vault (WrongVault 8010, not WrongAddress 8002). The code assertion pins F4 on its own. Evidence (tasks/forged-records/, run.sh: each mutant applied line-neutral, programs rebuilt, whole program suite run with the base's tests and then this commit's): F1 custody.rs:224 `if false` base SURVIVED, head KILLED F2 custody.rs:225 `|| false` base SURVIVED, head KILLED F3 custody.rs:226 `|| false && ..` base SURVIVED, head KILLED F4 lib.rs:306 `if false` base SURVIVED, head KILLED F5 lib.rs:323 `if false` base SURVIVED, head KILLED B1 custody.rs:64 `if false` base SURVIVED, head KILLED B2 custody.rs:65 `|| false` base SURVIVED, head KILLED B3 custody.rs:66 `|| false && ..` base SURVIVED, head KILLED (B1-B3 with 05a34c0's tests as the base: run-backing-r2.out. F4-F5 re-run after the control quote was added: run-f45-r3.out.) F1-F3 are killed by a_price_needs_the_protocols_own_asset_record_for_ that_mint, F4-F5 by protocol_and_vault_records_are_read_only_at_their_ own_addresses, and B1-B3 by a_backing_record_is_read_only_for_its_own_ protocol_owner_and_address. Re-run on 18b2e53 (r5/run.out): all eight base SURVIVED, head KILLED, by the same tests. Programs unchanged (no program source in this diff): Floor 86eb426e, Harbor 543be60b, Vault 7086148b at 18b2e53 and here (Floor was 8ca8a608 on r4's base). Sibling search: every derive/expect_pda in programs/vault/src. The only other re-derivation of a stored bump is the backing record (custody.rs:66). An earlier draft of this commit (05a34c0, never reviewed) said it already failed a test; run against B1-B3 that was false, so its three terms are tested here too. Every other site is expect_pda, which derives from known seeds before any write. Not searched here: Floor's authority_address (lib.rs:190) also re-derives a stored bump; whether a test pins it is a backlog item. docs/09: finding 113's stated gap names the two tests that fail it since 18 Oct. In the deletion-run row (Vault refusals deleted one at a time), the vault PDA survivor (lib.rs:324) moves from the unreachable guards to the tested ones: 19 of the 25 now fail a test, five of the other 6 are unreachable, and custody.rs:399 becomes the sixth. Finding 108's row keeps its 16 Oct "18 of the 25": it records what that fix did. The row for finding 120 itself comes in B's record unit for findings 117 to 124, which sits on this commit (claims.py wants section 4 gapless). Counts: program tests 176 -> 179 (Vault 116 -> 119). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Findings 117-119 and 121-124 (B: the guardian, its alerts and the emergency stop) and 120 (A: the Vault's record checks, tested by 4f1d294) were reserved on 18 Oct and fixed by units approved and stacked below this one, but none had a row in section 4. claims.py counts findings from those rows and wants them gapless, so they land here in one commit, in number order, on 120's unit. Each row is drawn from its unit's commit message: 117 86d2154; 118 7450491 and its sibling fixes f9e1959, 08f00bd, 5454d71; 119 1839935; 120 4f1d294 (the eight checks F1-F5, B1-B3 and its three tests); 121 c2eb320 and ec277bc; 122 8d4006c; 123 1c10cee; 124 18b2e53. The limits those messages state are in the rows (117: a wrong primary whose backup is down, a list `cluster`; 119: the survivors X5 and Z3; 121: async hooks; 122: Discord's /slack and /github). Some text comes from the code or a unit's evidence, not its message: 120, what a backing-term deletion does (its test asserts WrongAddress) and which code each copy gets; 121, that a hook error which cannot be made text leaves no line (delivery.test.mjs); 124, the cut lengths 120 to 300 (stop.mjs:76, main.mjs:372, :413) and that keys are not repaired (jsonText's replacer sees values); 119, "arithmetic, not run" for the two-minute figure (the unit's backlog entry). The "found by" cells follow each unit's row draft and backlog entry. Row 114's "not fixed here" for stop.mjs now names finding 117, and row 113's stated gap names finding 120. Dates are the reservation's, in UTC, as in rows 113 and 114: 117-124 were all reserved 18 Oct. README:178 and docs/12:22 count and list the findings: 116 -> 124, the range 96-116 -> 96-124, 120 joins the Vault tests and 117-119, 121-124 are listed as the guardian's, not yet on devnet. git grep for "116 findings" and "96[–-]116" found no other place. Evidence: python3 tools/claims.py: claims OK (26 files), findings 124. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An alert's detail is the reason, the line's message, then the parts it
names (change, proposal, program, mint) and `ticket N`. A failed send's
message is the RPC's JSON error, simulation logs included
(solana.mjs:301), and main.mjs:630 logs a failed cancel's error uncut.
webhookBody cuts Discord's body to 2000 units (Discord's message limit),
keeping the start, so a long error pushed out the parts and the ticket
the alert was about: the operator learned a cancel failed but not which
ticket.
Root cause: alertFor put the whole message in the detail; only the
Discord body bounded the total, from the start. Fix at alertFor: a
message over MESSAGE_MAX (1000) units is cut to its first 699, "…" and
its last MESSAGE_TAIL (300), with no half surrogate pair at either cut
(firstUnits, the Discord cut's helper renamed, and lastUnits). The start
holds the RPC's own message; the end holds the last logs (where a
program's failure is logged) and whatever a line says after an error it
quotes (main.mjs:111 "...; its supervisor must restart it"). named() is
given both pieces less the word at each cut, so a part said only in the
middle left out, or in a word a cut goes through ("change 4…" may be
change 45), is named again in full. The repeat key (first 60 units) and
the log line are unchanged. 1000 holds the guardian's own longest
messages (268 units, main.mjs:1403) whole, and leaves every part at its
longest within Discord's 2000: the test's worst failed line (a 240-unit
reason, keys of 44, ids of 2^64 - 1, a 44-key protocol in the title)
fits and ends with the ticket.
Evidence (tasks/alert-message-cap/):
- mutants.log: 17/17 KILLED by alert.test.mjs (no cut, i.e. the
parent's behaviour; start only; named on the uncut text, on the shown
text, on the start only; the word at either cut kept, or two words
dropped; half pair kept at either cut; cut at the cap; start one short
or one long; end one short; no ellipsis; cap 2000), restored rc 0.
- r2/mutants.log: the same 17/17 KILLED on this base, restored rc 0.
- node --test cloud/test/*.test.mjs: 251 pass, 0 fail.
Sibling search: Slack's text has no cap. A detail is now bounded by its
reason, 1000 units of message and its parts, so Slack's escape (at most
5x for & < >) stays far below Slack's documented 40,000 (not verified
against Slack). The JSON body carries the same cut detail: it was
already a summary (severity, title, detail, time), and the log line
keeps the whole text. Other log lines already cut their error
(stop.mjs: 120 to 160 units; main.mjs:917, 1242: 200). Left out
(backlog): a line with both msg and error alerts with msg only
(pre-existing); the Discord tail still relies on reasons and keys being
short (longest today: main.mjs:1242's, 215 units and the digits of its
seconds; every other reason is built from ids, keys and amounts,
policy.mjs); the repeat key's 60 units can merge two different long RPC
errors (pre-existing).
Finding 125: docs/09 §4 row; README and docs/12 counts 124 -> 125.
docs/07 says so; docs/09:40 250 -> 251.
A real refusal (tasks/alert-message-cap/r2/probe/: a local Agave 4.3
validator, a system transfer over the payer's balance) is 663 units,
its fields in key order: after `logs` come seven null fields (158
units), so the last 300 hold the last 142 units of logs, the 75-unit
"Program ... failed: custom program error: 0x1" line whole.
Code review (high, two passes) of this commit: the docs gave exact cut
lengths (699, 300) that a dropped half pair shortens by one: now said.
Kept as designed: the JSON and Slack bodies get the cut detail too
(named() needs one detail); the cap is fixed, not the room left in
Discord's body; a word at a cut is treated as cut even when it is whole,
so a part there may be named twice (noise, never a loss). Pass 2 also:
the repeat key's 60 units are the same for every simulation failure
(backlog).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
No core test caught `backed_amount` accepting health -1 (ledger.rs:246, `t >= 0` -> `t >= -1`): a ledger left 1 unit short. docs/09 said so in the ledger fuzz row; only the Python differential caught it. The property test g3_backed_amount_is_sound_and_maximal checks the same rule, but its random worlds seldom land on health exactly -1. g3_backed_amount_is_exact_on_small_ledgers enumerates two priced mints: other and own position in -6..=12 (0 is no position); per mint, (low, high) price in (1, 1), (2, 2), (3, 3), (1, 3) and haircut 5,000 or 10,000 with liability weight 10,000 or 15,000; request 0..=40; 3,789,056 cases. Each grant must equal the largest b <= request whose ledger owes nothing or has health >= 0, restated with collateral floored at its low price and haircut and debts ceiled at their high price and weight. In 764,280 cases some b is valid and one unit more leaves health at exactly -1; the test pins that count, and a Python restatement of the same domain prints 3789056 764280. Not covered, and said so: a full ledger, unpriced or unallowlisted mints, a price or weight of 0 (with hi = 0 the no-collateral shortcut grants less than the rule, docs/09 §2), other exponents or decimals. Tests only: no program or core source line moves. Evidence (tasks/g3-exhaustive/mutants.sh, mutants-r4.out): - M5 `t >= -1`: KILLED by the new test; the rest of the core suite (--skip the new test) passes on it, rc 0. - M6 `t > 0`, M7 collateral rounded up, M8 debt rounded down, M9 fast path without `!others_liab`, M11 collateral at the high price, M12 debt at the low price: KILLED. - M10 fast path `after > 0`: survives this test (with every mint priced, health then decides the same); g2_and_fail_closed_without_ prices kills it. - New test 0.27 s release, 4.09 s debug; the exhaustive suite 2.78 to 2.83 s in release (three runs), so 09 §5.2's heading says ~3 s in release, not ~6 s. - tools/claims.py OK (exhaustive 7); scripts/check.sh ALL CHECKS PASSED. Docs: docs/09 §2 and §5.2, README and docs/14 say 7 exhaustive tests over ~588M inputs (584.4M + 3.8M); 03 §4.6 cites the test for G3 (two priced mints, one position each) as it does g2's for G2; the ledger fuzz row says a core test now catches the -1 bound. Sibling search: docs/09 names no other mutant of a core rule that the core suite misses, apart from .cargo/mutants.toml's 22 exclusions (stated equivalents, §6). Those in ledger.rs (contribution, has_liability, backed_amount) concern zero positions and the health overflow fallback, which this domain never reaches; not re-run here. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
alertFor mutes a repeat of the same kind, a failing freeze for a
minute and anything else for 10 minutes, "per cause"; the kind held the
message's first 60 UTF-16 units. That held no cause (finding 126):
- Every refusal a node sends starts with the same 60. A local Agave 4.3
validator's (tasks/alert-message-cap/r2/probe/out.txt) begins
`sendTransaction: {"code":-32002,"message":"Transaction simul`, as
solana.mjs words every JSON-RPC error. A veto or cancel refused for a
second reason inside its window was muted.
- A freeze is sent without a preflight (signer.mjs sendEscalating), so
one the Vault refuses lands and fails, and signer.mjs throws
`transaction <signature> failed: {...}`. Each retry is signed anew, so
each was a new kind: a critical alert on every poll.
Root cause: the key was read from the text, when the error that made it
is structured. Fix: errorCause(e) in alert.mjs, logged by main.mjs as
`errorCause` on a failed action's line and a failed heartbeat's;
alertFor keys on it, else on the first 60 units as before. jsonText
drops it when undefined.
- A transaction's error is one cause however it came: a node's refused
preflight (rpcError.data.err), the transaction failing on chain
(onChainErr), or the freeze's own simulation (freezeRefusal, now in
policy.mjs so a test can make its error, carrying `simulatedErr`). An
instruction's error is only what it failed with: its index moves with
the compute-budget instructions, which the simulation leaves out.
- Any other RPC error is its method (Rpc.call now sets `rpcMethod`), its
code and its text with each run of digits as "#", in a word too
(behind by 42 slots is behind by 43 a poll later; "retry in 734ms"),
but a program's 0x1770 is kept.
- A signed transaction's failure (not confirmed, expired) is its message
without the signature, when the message holds it.
- A cause is cut at 160 units.
My own code review (high) of a first version (ee724da, on this branch
before the amend) found one Vault refusal under three keys (preflight,
on chain, simulation), the method dropped (a lagging node's read and
send were one kind), hex codes masked, and a primary send's network
error (signature set, not in its text) keyed on its whole text. A
second pass found digits joined to letters left unmasked ("734ms") and
the transaction-error causes uncut. Declined, with reasons: keying on a
structured `unconfirmed` flag instead of the message (a rewording of
send's error is caught by the test), a single place that attaches the
cause to every log line (the two failure sites are the ones that
alert), and the instruction index relative to the budget (stated below).
A third pass found no code defect: the test's on-chain failure now
really carries two budget instructions, alertFor's doc says every line's
kind holds its cause, and the data-less refusal limit is stated.
An
earlier version that parsed the JSON back out of the message is kept as
auto/B/alert-repeat-cause-r1, unsent.
Evidence (tasks/alert-repeat-cause/, r2/):
- New test "a failed action's alert repeats per cause", on errors made
by Rpc (stubbed fetch), send (fake endpoint, skipPreflight) and
freezeRefusal: 15/15. With the old key (mutant N1) it fails "another
cause within the minute"; with ee724da's errorCause, r2/red-r1.log.
- r2/mutants3d.log: 16/16 KILLED, each on its own assertion;
restored rc 0.
- A's review (CHANGES, F1, F2): the RPC code in a cause was unpinned,
and freezeRefusal, now exported, had no test of its own; a non-null
result ends the freeze's retries. New assertion (same method and text,
another code) and a new policy test: null for no error and for errors
it does not name; an error naming the error and carrying
`simulatedErr` for each fee, payer, rent and Vault refusal, Custom 0
included. r3/mutants2.log: 8/8 KILLED (A's A8, A1, A3; the payer and
Vault branches dropped, Custom 0 read as no code, the error left out
of the message, simulatedErr dropped), restored rc 0. A8 fails "the
same text, another code". A code review of this delta found the
Custom 0 and message gaps (fixed) and that the policy test repeats
two inputs of the alert test (kept: there they make causes).
- claims OK: findings 126, client 253.
Sibling search (`slice(0, N)` and message comparisons in cloud/): the
keeper (main.mjs keepApproved) logs a failed execute once per reason,
keyed on its first 160 units; one that lands and fails, or is not
confirmed, starts `transaction <signature>` and is logged every scan.
It is a warn that does not alert: backlog c37, not fixed here.
saveError compares the whole message; floorCheck and guardianSetCheck
key on their own codes and shapes. Other lines that wrap an error in
`msg` keep the first 60 units.
Not covered: no test runs main.mjs to a failed send, so its two
errorCause lines are checked by reading. One error from two
instructions of a transaction is one cause (a compute-budget
instruction and the Vault's failing alike), and so are two refusals
with one method and code, no transaction error, and texts that differ
only in digits or after 160 units. A text that changes in letters on
every poll (a hex request id) is still a new cause each time. A
refusal whose node sent no `data` keys on its text, so it is not one
cause with the same error from the chain or the simulation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The account-substitution sweep only ever offered real accounts: a fresh address, the transaction's own accounts, a second market's, programs and sysvars. None had the right bytes and the wrong owner, so an owner check was untested wherever nothing else refuses a forged account. Plant `Config::load(&acc(accounts, 0)?.try_borrow()?)` (no owner check) at one of Floor's 13 `load_config` calls at a time: 12 pass the whole Floor suite; only stop.rs:285 (close_buffer) fails, in the dedicated test. In the Vault, the same plant at any of its 12 owner-checked `load` calls passes the vault, e2e_world and differential tests. `Sweep::forgery(name, real)` adds a decoy that is a byte-for-byte copy of `real` owned by a program nobody here deployed (an attacker's), rewritten before each run so its state matches what the instruction expects (its last copy once `real` is closed), and offered from the first run where `real` exists. The copies go into each run's own states only, not the state it leaves behind (no test tells this; it keeps them out of the setup between runs). `Sweep::forged` lists the runs that name a forgery's real account at a slot swapped for decoys (checked, unsigned, no backstop) and were offered its copy; both tests pin that list, and `unused_forgeries` names any forgery no such run was offered. Forgery names must be unique. `for_market` and `new` now share one constructor. Floor's sweep forges its config; the Vault's forges the two queued tickets and the three changes, the accounts a Vault load checks by owner and recorded fields only, not by address. Evidence (tasks/sweep-repeated-accounts/; plants by matrix.sh and vmatrix.sh, run on the tests at 804e297 (matrix3.out, vmatrix1.out) and on this commit (r5-matrix.out, r5-vmatrix.out)): - Floor: 11 of the 12 sites the old tests pass now fail the sweep; stop.rs:285 fails it too. abort (stop.rs:104) still passes: its spill must equal its config (:134), which refuses the copy at either slot alone. The next commit covers that. - A copy taken once at setup instead of before each run (matrix.out) catches only 5 of those 11: stage_abort, restore_snapshot, veto, approve, execute and cancel refuse a stale copy on its state. - Vault: the old tests pass all 12; this commit fails the four of tickets (brakes.rs:37, :77, custody.rs:499) and changes (governance.rs:410). The other eight load PDAs whose address the same instruction checks (lib.rs:304, :318, custody.rs:63, :223, genesis.rs:314, :348, governance.rs:145, :269): a copy elsewhere is refused with or without the owner check. - The Vault's owner check weakened to refusing only system-owned accounts (lib.rs:182, `if a.owned_by(&SYSTEM_PROGRAM)`): with system-owned copies (this commit's first form) the vault, e2e_world and differential tests pass (weak-r4c1sys.log); with copies under another program the sweep fails at all six ticket and change runs (weak-r5c1.log). - With a check skipped the runtime refuses the copy where the program writes it or takes its lamports (16 ExternalAccountDataModified, 4 ExternalAccountLamportSpend); the sweep reports that as not the program's own code. Three swaps went through: close_buffer (only reads the config), a cancel of an already vetoed change and a heartbeat (both write back the bytes they read: the runtime refuses only a change to them). - Pins: a forgery of a change 3 that never exists: left ["change 3, forged"] (r4-stale-mutant.log); of the SOL vault's ticket, which no run names: left ["sol ticket"] (r4-unnamed-mutant.log, before the rename); a name registered twice panics (r5-dup-mutant.log). Counting a free slot as swept survives (r5-free-mutant.log): no forged account stands only at a free slot in any run. Nor does one stand at a signer or backstop slot, so the "unsigned" and "no backstop" parts of the rule for `forged` have no case either. - Survivor: a copy taken once, when its account first exists, and never refreshed passes on the real programs. Only a plant tells it, by the matrix.out run above. - Green: 5,623 Vault and 1,782 Floor substitutions (before 5,365 and 1,710), 673 and 171 at backstop slots (673 and 163). cargo test --release in programs/tests: all pass (r6-c1-all.log); claims OK. sBPF unchanged (tests only): Floor 86eb426e, Harbor 543be60b, Vault 7086148b. Sibling search: every `owned_by(program_id)` in Floor and the Vault (Floor lib.rs:178; Vault lib.rs:182, through `load`); Harbor holds no state of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep swaps a key that signs nowhere one slot at a time. Where an
instruction checks that two of its slots are equal, that check refuses
every such swap first and hides any missing check of the account
itself. Floor's abort is one: it loads its config from slot 0 and
requires the spill (slot 5) to be the same account (stop.rs:134,
finding 88). With the config's owner check skipped there
(`Config::load(&acc(accounts, 0)?.try_borrow()?)` at stop.rs:104), the
whole Floor suite passes, with or without the previous commit's forged
copy.
Now each key that signs at no slot and stands at two or more checked
slots, none with a backstop, is also swapped at all of them together,
and at its free slots, so it is the same throughout, for the same
candidates as a one-slot swap (now one helper, `candidates`, for both,
and one, `allows`, for the allowed substitutes). A key with a backstop
at one of its slots is skipped: every such swap changes that slot, so
the backstop's refusal would excuse it whichever check is missing.
`Sweep::together` lists the keys swapped this way, once a swap was sent.
Floor's test pins its one (the abort's config); the Vault's pins none:
registration by the caller PDA names it to Harbor (slot 1, a backstop:
Harbor's seeds check) and to the Vault (slot 6).
Evidence (tasks/sweep-repeated-accounts/):
- abort plant: the previous commit passes all 60 Floor tests
(r6-c1-abort-plant.log); this one fails the sweep: "abort (stop):
account 0 replaced by A's config, forged at [0, 5]: refused by
ExternalAccountDataModified, not the program's own code"
(r6-plant.log).
- M2 (swap only the first of its slots) + the plant: both sweeps pass
(r6-M2-plant.log). M2 survives on the real programs; only a planted
bug tells it, as for the sweep's other tries.
- M1 (`slots.len() < 3`, never runs): Floor's pin fails, left []
(r6-M1.log). M3 (backstop skip removed): the Vault's pin fails, left
[("register (caller PDA)", ..)] (r6-M3.log). The sent-only record has
no case here (no run allows every candidate at all of a key's slots):
no test tells it.
- Green: 5,623 Vault and 1,804 Floor substitutions (5,623 and 1,782
before: the one-slot swaps are unchanged by the shared helper), 673
and 171 at backstop slots. cargo test --release in programs/tests:
all pass (r6-c2-all.log); claims OK. sBPF unchanged: Floor 86eb426e,
Harbor 543be60b, Vault 7086148b.
Sibling search: a probe in Sweep::run listing every run with a
non-signing key at two or more non-free slots found only these two (in
all three sweeps). Signing keys at several slots are still swapped one
slot at a time (backlog 0v).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two commits before this one close a gap of finding 113's kind: the owner check of Floor's config and of the Vault's tickets and changes was skipped at 16 loads without a test that could fail (11 Floor config loads and the abort's at stop.rs:104, and the Vault's ticket loads at brakes.rs:37, :77, custody.rs:499 and its change load at governance.rs:410). 113, a sweep blind spot with no program defect, has a number and a §4 row; so does this one (B's review F1). docs/09 §4 gets row 127, and the "Account substitution" row cites it. README and docs/12 count 127 findings, 127 among the fixed (Floor and Vault tests). No code changes. The row's "Not covered" names what only a planted bug tells (the refresh and the swap at every slot of a key: B's S6 and the second commit's M2), what has no case (the together swap's `all` and free-slot rules, B's S4, S5; the three qualifiers of `forged`, B's S1, S2 and my free-slot mutant; the sent-only record; copies kept out of a run's leftover state), and that the copies are listed by hand (backlog: forge every program-owned account). It also says that most of the 16 fail the sweep only by its own-code rule. The §2 row said the copies were "copied afresh before each transaction"; they are brought up to date while the account exists, and its last copy stays on offer once it is closed (af1da40's message). Fixed here, beside the new cites. Evidence: - The plants re-run on this branch (tasks/sweep-repeated-accounts/r2/ matrix-all.sh, matrix-all.out): the owner check skipped at each of the 17 loads no address check pins, one at a time, programs rebuilt, and the whole programs/tests suite (cargo test --release --no-fail-fast; two generators are #[ignore]d) run with the tests of ee724da and of this commit (the run's head variant; its three test files are byte-identical to this commit's, cmp, as the job ran on an earlier draft of this commit that differs only in docs/09). The Vault's eight address-pinned loads were not re-run here; their earlier runs are in the first commit's message. ee724da: 16 pass, stop.rs:285 fails in floor_reads_only_a_config_it_wrote_and_a_buffer_the_loader_owns. This commit: all 17 fail the sweep (every_floor_account_substitution _is_refused for Floor's 13, every_account_substitution_is_refused for the Vault's 4). sBPF after the run as before: Floor 86eb426e, Harbor 543be60b, Vault 7086148b; git status clean. - python3 tools/claims.py: claims OK, findings 127. - B's review of 273b129 (the same two patches on fb44f95) re-ran three plants (governance.rs:410, stop.rs:104, :285) and checked every line cite against the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main.mjs logs some failures as what went wrong in `msg` and the RPC's words in another field: an RPC failover (`cause`), a node refusing the compute budget (`refused`), the freeze falling back without its budget and a startup replay skipping a transaction (`error`). alertFor's text was `line.msg ?? line.error`, so these alerts, all critical, said what failed but not why: "RPC endpoint a is failing; now using b". Finding 125's row states it as not covered (finding 128). Root cause: alertFor took one text per line, and the lines named their cause three ways. Fix: main.mjs's two odd fields are now `error`, and alertFor's detail shows `msg | error` when a line has both, cut as one text at MESSAGE_MAX, so a long error keeps its end and the parts after it. A line's urgency and its repeat key stay on its message, as before: the RPC's words are not the guardian's urgent warnings, and their figures change between polls (a failover with a new cause inside 10 minutes is still muted). An empty msg or error counts as absent in the detail, the key and the urgency alike. Evidence (tasks/alert-line-error/): - New test "a line with a message and an RPC's error alerts with both": red on the old code (red.log: the detail without the error). It also reads main.mjs: every field valued from an error's `.message` is named `error` (6 found). restart.test.mjs's failover at start now checks the real line's `error` and its alert; no test reaches the other three lines. - mutants4.log: 13/13 KILLED (detail on msg only, an empty or an absent part kept, parts heard in msg only, separator, key on the joined text, urgency on the joined text, error first, cut on msg only, an empty msg keyed, main.mjs's `cause` or `refused` back, and `cause` against restart.test.mjs alone); restored rc 0. - node --test: 254/254. claims OK: client 254; docs/09:40 says 254. - The scan's comment says what it checks: a field whose value is a bare `.message`, maybe through `String(`; a template or `String(e)` escapes it (A's review, N1). My code review (high, three passes) found an empty error shown as "msg | ", an empty msg keyed on "" while its error was shown, the renames untested (and a test that already starts main.mjs to a failover), and two derivations of the text; all fixed. Declined, with reasons: keying these lines on errorCause (their `msg` is the guardian's own why, and finding 126 left lines other than a failed action's on their first 60 units); a program key in quoted logs stopping "program X" from being named (no line with both a message and an error has a program or mint); renaming the stop's failover `cause` too (the stop sends no alerts, and its log is what an operator reads during a stop); another separator than the detail's " | " (the error is one more part of the detail, as a failed action's is); and cutting the error apart from the message (these messages are 64 to 164 units in the source, about 540 with two 253-unit hosts, under the 699 the cut keeps, so the error's start is always shown). A fourth pass found no code defect: an empty msg now keys on its error (no line logs an empty msg), and a crossCheck switch's reason is in both of its alerts (the failover and the warn after it: two alerts per switch, as before). Sibling search (log calls in cloud/guardian with an error in a field other than msg or reason): main.mjs 138, 372, 398, 399, 903; 398 is a warn that does not alert. stop.mjs logs `error` and `cause` beside `msg`, but sends no alerts. Not covered: main.mjs cuts these errors to 160-300 units before they are logged, so their end (the last simulation logs) never reaches the alert; and a failover keys on its first 60 units, which a long first host name fills, so a second failover to another endpoint is muted (both as before; backlog). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit before this one fixes the alert of a line that has both a message and an RPC's error: the alert showed the message alone, so an RPC failover, a refused compute budget, the freeze sent again without its budget and a skipped startup replay said what failed but not why. Finding 125's row states it as not covered; this gives it a number. docs/09 §4 gets row 128. README and docs/12 count 128 findings, 96–128 recorded after the submission, and 128 among the guardian's fixed. No code changes. The row's "Not covered" names the three lines no test reaches, main.mjs's own cut before logging (160, 200 or 300 units, by line) and the failover's repeat key on its first 60 units (finding 129's subject). Evidence: - python3 tools/claims.py: claims OK, findings 128, client 254. - The row was drafted with the fix (tasks/alert-line-error/row-128.md) and is added after A's row 127 (e313bf4). My code review of this record found its "160–300 units ... never reaches the alert" loose: it now names each cut (main.mjs:141 160, :907 200, :376 and :403 300) and says only a longer error loses its end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An RPC failover logs `RPC endpoint <from> is failing; now using <to>`, and alertFor keyed a line with no subject and no errorCause on its message's first 60 units. From a 24-unit from-host on (solana-mainnet.g.alchemy.com is 28), <to> starts past unit 60, so a move a -> b and then a -> c within the 10-minute window was one kind and the second was muted: the operator was not told the guardian now reads from c (finding 129). The message also named hosts only, so two entries on one host (FailoverRpc: "a URL listed twice does not name its entry") read the same. Fix: the failover line names both endpoints by entry, as the cluster lines do (endpointName: `host (rpc)`, `host (rpcs[0])`), and in fields, `endpointFrom` and `endpoint`, which about() counts as parts of the subject, like a change, program or mint. The same move still repeats once a window. The parts are matched bare, so the detail does not name them again when the message says them; a message cut at MESSAGE_MAX names them at the end, as for the other parts. Tests: - alert.test.mjs "a failover to another endpoint is news even when its host falls past the key's 60 units (finding 129)": a->b, a->b muted, a->c alerts with the detail unchanged, a->d (c's host, another entry) alerts, c->b alerts, and two moves to b from two entries on one 49-unit host both alert (their names differ past unit 60). - restart.test.mjs (finding 114 start failover): the real line reads `RPC endpoint <host> (rpc) is failing; now using <host> (rpcs[0])` and carries endpointFrom/endpoint = `<host> (rpc)`/`<host> (rpcs[0])`. - tasks/failover-key/mutants3.log: 38dc10f's code with the new tests RED; 9/9 mutants KILLED (no endpoint part, no from part, either not bare; line without either field; endpoint = from; from = to; both by host); restored GREEN. - scripts/check.sh: ALL CHECKS PASSED (tasks/failover-key/check.log, before the from field; check3.log on this commit, check-exit=0). - docs/09:40 client tests 255; tools/claims.py OK, client 255. Code review (high, three passes): fixed the entry naming, the from-entry past unit 60, three comments and a doubled hostOf. Declined: keying every line on its whole message with digits normalised (lines quoting a signature or mint would then never repeat as one kind). Not fixed here, backlog c42: moves a->b, b->a, a->b within the window mute the third, so the last alert names the endpoint the guardian left; never muting a move trades that for one alert per switch when two endpoints flap. Backlog c43: a cross-check move (crossCheck, stale endpoint) logs the same "is failing" line; its reason is logged on its own warn line (main.mjs:1488). Sibling search (main.mjs, every error/warn line with no errorCause): the other lines whose distinguishing words sit past unit 60 carry a subject field (changeId 735, ticketId 624, program 875, mint 340) or start with what differs (1232, 1334). Not fixed here, backlog c44: the wrong- cluster line (147) names its entry at unit 13, but one host of 40+ units listed twice puts the entry's index past unit 60, and onCluster says it once per endpoint, so the second is never heard; no test reaches line 147 today. stop.mjs's failover line sends no alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit before this one keys a failover's repeated alert on the
endpoints it moved from and to. Its line had no subject, so alertFor
keyed on its first 60 units, which held only the first 24 units of the
new endpoint less the first host's length: two moves from one endpoint
to hosts that begin alike were one kind and the second was muted, and
from a 24-unit first host on every move from it was. This gives it a
number.
docs/09 §4 gets row 129, and row 128's limit on the failover's key
points to it. README and docs/12 count 129 findings, 96–129 recorded
after the submission, and 129 among the guardian's fixed. No code
changes.
The row's "Not covered" names the a -> b, b -> a, a -> b case (the
third is muted, so the last alert names the endpoint the guardian
left), a move off an endpoint that answers (crossCheck's behind or
missing recent transactions, call()'s wrong cluster: the line says "is
failing", its error says why; backlog c43), and the
wrong-cluster line's like key from a 40-unit host (backlog c44).
Evidence:
- Units (node): "RPC endpoint " 13, " is failing; now using " 23,
"solana-mainnet." 15, "solana-mainnet.g.alchemy.com" 28; "RPC
endpoint " + a 40-unit host + " (rpcs[" is 60, so the index digit is
unit 60 on. main.mjs:133 names the primary `host (rpc)`.
- The moves that are not failures: solana.mjs:495 ("RPC endpoint is
behind"), :528 ("missing recent transactions"), :562 ("serves
another cluster"); each goes through switchTo -> onSwitch, which
logs main.mjs:141's line.
- python3 tools/claims.py: claims OK, findings 129, client 255.
My code review (high) of the record found the collision stated only
for a long first host, row 128 not pointing here, "Not fixed" where
every other row says "Not covered", the crossCheck move left out, the
example name `host (rpcs[0])` that misses the primary's `(rpc)` (also
in alert.mjs's comment, fixed in the commit before), the wrong-cluster
threshold left vague (and given as 36 in an earlier draft of the
commit before; 40 is right), and row 128's "160–300 ... never" (fixed
in row 128's commit). All fixed. A second pass found the 15-unit
example's own condition (a 9-unit first host) unstated, "has no
subject" in the present tense, and crossCheck's second move and
call()'s wrong-cluster move left out: fixed. It also asked to give the
wrong-cluster line its `endpoint` field here; declined: no test reaches
main.mjs:147, so that fix needs its own test and commit (backlog c44).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep offered a copy under another owner only of the accounts a test named (Floor's config, the Vault's tickets and changes; row 127's first "Not covered" item). Every other account the swept program owns, the protocols, vaults, backing and asset records and Floor's third config, was offered as itself or as another real account, never as the same bytes at another address under another owner. So a load whose owner check and address check were both gone still passed the sweep: every other decoy differs in its bytes and is refused by a later check. Now each account of the swept program a transaction names, unless a forgery already names it, gets a copy the same way (brought up to date before each run, its last copy kept once it closes). The swept program is a field of the sweep (the Vault program, or Floor), not the program the transaction calls: Harbor's deposit, withdrawal, payout and registration reach the Vault by CPI. Named forgeries stay: the tests pin the runs offered each, and naming an account the sweep has already copied fails. The new `copied` list records the runs offered a copy at a swept slot, and the tests pin it: Floor's is exactly the third config at "execute release to none"; in the Vault's first world the copied accounts are exactly the protocol, the USDC vault, its asset record and the whale's backing (56 runs), and in the seal's world exactly the protocol, the USDC asset record an upsert rewrites and the three vaults the seal reads (11 runs). A scratch mutant of the seal-world pin (the SOL vault swapped for the SOL asset in the expected set) fails it at vault.rs:3477 (seal_pin_mutant.sh, .log). Evidence (tasks/sweep-signers/ in the autopilot dir): - Probe: 53 (run, slot) pairs held an account of the called program at a swept slot with no copy (probe3.log; it missed Harbor's calls, as the first version of this change did, which code review caught). - Owner and address check of one load both skipped, programs rebuilt (doubles.py, doubles2.out): the protocol (lib.rs:304, :306), a vault (:318, :323), a backing record (custody.rs:63, :66), an asset record in the price table (:223, :226). Each: old sweep SURVIVED, new sweep KILLED (a heartbeat, an attest, a quote, a backlog reset, a seal, withdrawals and admin payouts took a copy; doubles3.out). Each was already KILLED by the whole suite through its own test (doubles-full.out), so this is not a finding. - Mutants of the new rule (rule_mutants.py, rule_mutants4.out): no copies (N1), copies of any owner's accounts (N2), a copy where a forgery names the account (N3), copies recorded as named (N5), copies of the called program's accounts instead of the swept one's (N7): KILLED. Copies made but never offered (N6): SURVIVED; told only by the planted bugs above, as row 127 says of a copy never refreshed. The "one copy per account" assert has no case in these tests (no test names an account after a run named it). - Counts: 5,623 -> 6,264 Vault and 1,804 -> 1,808 Floor substitutions; 673 -> 745 and 171 -> 172 at a backstop slot (base measured here). The Vault sweep test ran in 0.82 s, against 0.74 s before. - Programs unchanged: sBPF floor 86eb426e, harbor 543be60b, vault 7086148b before and after. Also measured: no swept transaction has a signing key at two or more checked slots (Harbor's deposit and withdrawal have one at two slots, one of them free), so row 127's "signing keys at several slots" item has no case today. docs/09 §2 says so. Sibling search: the twin sets are listed by hand too (unused_twins fails a stale set, not a missing one), as are the extra decoys and keys each test passes; not changed here. Accounts owned by other programs are not copied (N2 shows executable accounts cannot be set, so that needs per-kind handling; out of scope). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
B's review of 2a2227c found two behaviours of the sweep's own copies that the docs/09 row states but nothing tells: a copy never brought up to date after first sight (B's M2) and one dropped once its account closes (M3). Both mutants survive both sweeps. The row listed only a copy made but never offered as unpinned. Measured, not argued (tasks/sweep-copy-fix/): - A probe printing each refresh of a sweep-made copy whose data or lamports changed, and each run where its account no longer exists: Vault 32 changes over 6 accounts, 0 closes; Floor 0 and 0 (probe-vault.log, probe-floor.log). Probe reverted. - The four planted double mutants (owner and address check of a protocol, vault, backing and asset load skipped) with M2 applied to sweep.rs: all four still fail the sweep (doubles_m2.out: P1, V1, B1, A1 head sweep KILLED; rebuild rc 0, hashes 86eb426e, 543be60b, 7086148b). So M2 has no case among them and M3 none in today's runs. The row now says both are told by nothing, with these numbers. Also B's nit: the Vault's runs offered a copy are pinned by count, so the row says "how many runs were offered one". Docs only; no program or test change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A backup that does not answer getGenesisHash at start is asked again before each freshness check. If it then answers from another cluster, main.mjs logs `RPC endpoint <host> (rpcs[i]) serves another cluster than <cluster> ...`, once per endpoint (onCluster reports a hash when first learned). The line had no subject field and no errorCause, so alertFor keyed its repeats on its message's first 60 units. The entry's index sits at unit 20 plus the host's length, so from a 40-unit host two entries on that host were one kind, and the second entry's wrong cluster was muted for 10 minutes and never said again (finding 130). Fix: the line names its entry in the `endpoint` field, as the failover line does since finding 129; about() counts it as a part of the subject, matched bare, so the detail does not name it twice. Tests: - restart.test.mjs "a backup silent at start that later answers from another cluster is named in a field, which keys its alert": a started guardian whose backup gives no hash at the start's check and testnet's at the first freshness check (freshnessCheckS 10, the lowest checkConfig allows; ~10.5 s, killed at 60 s). It reaches main.mjs:148, which no test reached before, and checks the line's `endpoint` and message, that it came at least 9 s after the start's silent line (so at the freshness check, not by a call moving to the backup), and that the real line keys its alert: it repeats once a window, and the same line naming another entry alerts. Red on the old code: `actual: undefined, expected: '127.0.0.1:62715 (rpcs[0])'` (tasks/cluster-line-key/red-restart.log). - alert.test.mjs "a backup's wrong cluster is news for each entry, even past the key's 60 units": entries rpcs[0] and rpcs[1] on a 48-unit host both alert; the same entry repeats once a window; the detail is the message alone. - tasks/cluster-line-key/mutants3.log: 7/7 KILLED (no field; field by host; field naming the primary; field renamed `entry`; about() without the endpoint part; the part not bare; crossCheck not asking an unchecked endpoint again), restored green. - cd cloud && node --test: 257/257. docs/09:40 client 257; python3 tools/claims.py: claims OK, client 257. Code review (high): fixed the test's 20 s kill against a 10 s check on a slow runner, its not showing when the hash was learned, the unit test's line built by hand (the real line now keys an alert in the restart test), and the entry's name built twice. A second pass: the kill could come inside the next log line, which the test then parsed (it now waits for the line's end and parses whole lines only), and two test comments said more than their tests show. Declined: keying every line on its whole message (declined in finding 129's commit: lines quoting a signature or mint would never repeat as one kind); stop.mjs's lines, which send no alerts; the test's 10 s (the line is main.mjs's, so a started guardian is the only way to reach it). Backlogged (c45): a silent backup that later answers on the right cluster is an info line, so the start's error alert is never followed by one saying it is in use. Sibling search: main.mjs's other error lines are as finding 129's commit found them (a subject field, or what differs first). The cluster's "answers on <cluster>" line (149) is info and does not alert. stop.mjs's cluster and failover lines send no alerts. The start's wrong-cluster and silent lines (1331, 1335) are one line each, listing every entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit before this one gives the wrong-cluster line, logged when a backup silent at the start later answers from another cluster, the `endpoint` field its alert keys on. Its message keyed on its first 60 units put the entry's index past them from a 40-unit host, so a second entry on that host was muted. This gives it a number. docs/09 §4 gets row 130, and row 129's last limit points to it in the past tense. README and docs/12 count 130 findings, 96–130 recorded after the submission, and 128–130 among the guardian's fixed. No code changes. Evidence: - Units (node): "RPC endpoint " + a 48-unit host + " (rpcs[" puts the index digit at unit 68 = 20 + 48 (tasks/cluster-line-key). - python3 tools/claims.py: claims OK, findings 130, client 257. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shuv-amp
force-pushed
the
auto/B/cluster-line-key-r2
branch
from
October 19, 2026 17:55
4d08742 to
960c01d
Compare
This was referenced Oct 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Off-chain guardian (cloud/guardian/main.mjs) and its tests, plus docs (docs/09, README, docs/12). No program change. Two commits.
endpoint: endpointName(at), which the repeat key already reads, so each entry alerts on its own.Proof
endpointundefined) and passes with the fix (autopilot tasks/cluster-line-key/red-restart.log).alert.test.mjspins the repeat-key side with a 48-unit host.warn,about()keying the endpoint only in some cases, the freshness check made immediate) were also KILLED.alertFor: the index digit falls at 20 + host length, so a 39-unit host still alerts and a 40-unit host is muted.scripts/check.shALL CHECKS PASSED.Limits
🤖 Generated with Claude Code