Repository navigation
Conversation
An RPC failover logs `RPC endpoint <from> is failing; now using <to>`, and alertFor keyed a line with no subject and no errorCause on its message's first 60 units. From a 24-unit from-host on (solana-mainnet.g.alchemy.com is 28), <to> starts past unit 60, so a move a -> b and then a -> c within the 10-minute window was one kind and the second was muted: the operator was not told the guardian now reads from c (finding 129). The message also named hosts only, so two entries on one host (FailoverRpc: "a URL listed twice does not name its entry") read the same. Fix: the failover line names both endpoints by entry, as the cluster lines do (endpointName: `host (rpc)`, `host (rpcs[0])`), and in fields, `endpointFrom` and `endpoint`, which about() counts as parts of the subject, like a change, program or mint. The same move still repeats once a window. The parts are matched bare, so the detail does not name them again when the message says them; a message cut at MESSAGE_MAX names them at the end, as for the other parts. Tests: - alert.test.mjs "a failover to another endpoint is news even when its host falls past the key's 60 units (finding 129)": a->b, a->b muted, a->c alerts with the detail unchanged, a->d (c's host, another entry) alerts, c->b alerts, and two moves to b from two entries on one 49-unit host both alert (their names differ past unit 60). - restart.test.mjs (finding 114 start failover): the real line reads `RPC endpoint <host> (rpc) is failing; now using <host> (rpcs[0])` and carries endpointFrom/endpoint = `<host> (rpc)`/`<host> (rpcs[0])`. - tasks/failover-key/mutants3.log: 38dc10f's code with the new tests RED; 9/9 mutants KILLED (no endpoint part, no from part, either not bare; line without either field; endpoint = from; from = to; both by host); restored GREEN. - scripts/check.sh: ALL CHECKS PASSED (tasks/failover-key/check.log, before the from field; check3.log on this commit, check-exit=0). - docs/09:40 client tests 255; tools/claims.py OK, client 255. Code review (high, three passes): fixed the entry naming, the from-entry past unit 60, three comments and a doubled hostOf. Declined: keying every line on its whole message with digits normalised (lines quoting a signature or mint would then never repeat as one kind). Not fixed here, backlog c42: moves a->b, b->a, a->b within the window mute the third, so the last alert names the endpoint the guardian left; never muting a move trades that for one alert per switch when two endpoints flap. Backlog c43: a cross-check move (crossCheck, stale endpoint) logs the same "is failing" line; its reason is logged on its own warn line (main.mjs:1488). Sibling search (main.mjs, every error/warn line with no errorCause): the other lines whose distinguishing words sit past unit 60 carry a subject field (changeId 735, ticketId 624, program 875, mint 340) or start with what differs (1232, 1334). Not fixed here, backlog c44: the wrong- cluster line (147) names its entry at unit 13, but one host of 40+ units listed twice puts the entry's index past unit 60, and onCluster says it once per endpoint, so the second is never heard; no test reaches line 147 today. stop.mjs's failover line sends no alerts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit before this one keys a failover's repeated alert on the
endpoints it moved from and to. Its line had no subject, so alertFor
keyed on its first 60 units, which held only the first 24 units of the
new endpoint less the first host's length: two moves from one endpoint
to hosts that begin alike were one kind and the second was muted, and
from a 24-unit first host on every move from it was. This gives it a
number.
docs/09 §4 gets row 129, and row 128's limit on the failover's key
points to it. README and docs/12 count 129 findings, 96–129 recorded
after the submission, and 129 among the guardian's fixed. No code
changes.
The row's "Not covered" names the a -> b, b -> a, a -> b case (the
third is muted, so the last alert names the endpoint the guardian
left), a move off an endpoint that answers (crossCheck's behind or
missing recent transactions, call()'s wrong cluster: the line says "is
failing", its error says why; backlog c43), and the
wrong-cluster line's like key from a 40-unit host (backlog c44).
Evidence:
- Units (node): "RPC endpoint " 13, " is failing; now using " 23,
"solana-mainnet." 15, "solana-mainnet.g.alchemy.com" 28; "RPC
endpoint " + a 40-unit host + " (rpcs[" is 60, so the index digit is
unit 60 on. main.mjs:133 names the primary `host (rpc)`.
- The moves that are not failures: solana.mjs:495 ("RPC endpoint is
behind"), :528 ("missing recent transactions"), :562 ("serves
another cluster"); each goes through switchTo -> onSwitch, which
logs main.mjs:141's line.
- python3 tools/claims.py: claims OK, findings 129, client 255.
My code review (high) of the record found the collision stated only
for a long first host, row 128 not pointing here, "Not fixed" where
every other row says "Not covered", the crossCheck move left out, the
example name `host (rpcs[0])` that misses the primary's `(rpc)` (also
in alert.mjs's comment, fixed in the commit before), the wrong-cluster
threshold left vague (and given as 36 in an earlier draft of the
commit before; 40 is right), and row 128's "160–300 ... never" (fixed
in row 128's commit). All fixed. A second pass found the 15-unit
example's own condition (a 9-unit first host) unstated, "has no
subject" in the present tense, and crossCheck's second move and
call()'s wrong-cluster move left out: fixed. It also asked to give the
wrong-cluster line its `endpoint` field here; declined: no test reaches
main.mjs:147, so that fix needs its own test and commit (backlog c44).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shuv-amp
added a commit
that referenced
this pull request
Oct 19, 2026
…es (#138-#147) Batched squash of the reviewed, stacked PRs #138-#147 (one tree, identical to the reviewed stack's; each PR reviewed adversarially and green in CI on its own head): - #138 Test the Vault's record checks with forged records (finding 120) - #139 Record findings 117 to 124 in docs/09 section 4 - #140 Cut a long RPC error in an alert so the ids after it reach Discord (finding 125) - #141 Pin G3's health bound at its edge with an exhaustive test - #142 Key a failed action's repeated alert on its error's cause (finding 126) - #143 Offer the sweep forged owner-checked accounts, swapped at every slot (finding 127) - #144 Show an RPC's error in an alert whose line has a message (finding 128) - #145 Key an RPC failover's alert on the endpoints it moved between (finding 129) - #146 The sweep copies every account of the swept program, not only named ones - #147 Key a backup's wrong-cluster alert on its endpoint entry (finding 130) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Finding 129 (off-chain, guardian alerts). Two commits: 45e3f85 (the fix) and 226c20e (row 129). No program or core file changes.
A second RPC failover inside 10 minutes could be muted. When that happened, the operator was not told which endpoint the guardian had moved to. A failover line reads
RPC endpoint <from> is failing; now using <to>. It had no subject and noerrorCause, so alertFor keyed its repeats on the line's first 60 UTF-16 units. The new endpoint starts at unit 36 plus the first host's length. So the key held only the new endpoint's first (24 minus first host's length) units:solana-mainnet.(15 units) were one kind.solana-mainnet.g.alchemy.comis 28), every move away from that host was one kind.The line also named hosts only, so two entries on one host read the same.
The fix: the line names both endpoints by their entry (
host (rpc),host (rpcs[0])), never by URL, since a URL may hold an API key. It also logs them in two fields,endpointFromandendpoint, and alertFor counts these as parts of the subject, as it does for a change or a mint. The same move still repeats once a window.docs/09: row 129, and the client test count goes to 255. Row 128's failover limit now points to 129. The README and docs/12 finding ranges now end at 129.
Proof
bin/integrate.shran each-commit, green on 45e3f85 and 226c20e.scripts/check.shon autopilot/integration at 226c20e: ALL CHECKS PASSED; claims OK, client 255, findings 129 (autopilot tasks/review-rows-128-129/check-226c20e.log).Limits
crossCheckfinds it behind or missing recent transactions, or it serves another cluster. These log the same "is failing" line, and its error says why (B's backlog c43).🤖 Generated with Claude Code