Skip to content

Key an RPC failover's alert on the endpoints it moved between (finding 129) - #145

Closed
shuv-amp wants to merge 2 commits into
auto/B/alert-line-error-r3from
auto/B/failover-key-r2
Closed

shuv-amp wants to merge 2 commits into
auto/B/alert-line-error-r3from
auto/B/failover-key-r2

Conversation

@shuv-amp

Copy link
Copy Markdown
Owner

Opened by the Bulkhead autopilot agents: one reviewed unit of work. Every commit was reviewed adversarially by the other agent, with mutants, and passes tools/claims.py on its own.
Stacked on #144: merge that first; this PR is then rebased onto main.
The agents never merge. The owner reviews and merges.

What changed

Finding 129 (off-chain, guardian alerts). Two commits: 45e3f85 (the fix) and 226c20e (row 129). No program or core file changes.

A second RPC failover inside 10 minutes could be muted. When that happened, the operator was not told which endpoint the guardian had moved to. A failover line reads RPC endpoint <from> is failing; now using <to>. It had no subject and no errorCause, so alertFor keyed its repeats on the line's first 60 UTF-16 units. The new endpoint starts at unit 36 plus the first host's length. So the key held only the new endpoint's first (24 minus first host's length) units:

  • From a 9-unit first host on, two moves to hosts that share solana-mainnet. (15 units) were one kind.
  • From a 24-unit first host on (solana-mainnet.g.alchemy.com is 28), every move away from that host was one kind.

The line also named hosts only, so two entries on one host read the same.

The fix: the line names both endpoints by their entry (host (rpc), host (rpcs[0])), never by URL, since a URL may hold an API key. It also logs them in two fields, endpointFrom and endpoint, and alertFor counts these as parts of the subject, as it does for a change or a mint. The same move still repeats once a window.

docs/09: row 129, and the client test count goes to 255. Row 128's failover limit now points to 129. The README and docs/12 finding ranges now end at 129.

Proof

  • A new test fails on the old code. The started guardian's real failover line is checked. B's mutants: 9/9 killed (autopilot tasks/failover-key/mutants3.log).
  • Agent A's review of d2a1ef3 (REVIEWS/auto-B-failover-key.md): APPROVE. On the old alert.mjs and main.mjs the new tests fail (2 red). A ran 10 mutants of its own (A1-A5, M1-M5): 10/10 killed.
  • -r2.md (226c20e): 45e3f85's code diff from d2a1ef3 is the alert.mjs comment only. A re-checked the row's unit arithmetic in node: 13, 23, 15 and 28 units, and with a 40-unit host the index digit falls at unit 60. A also checked the three non-failure moves (solana.mjs:495, :528, :562). Each-commit green x4. APPROVE, with two nits about wording.
  • At integration (19 Oct): bin/integrate.sh ran each-commit, green on 45e3f85 and 226c20e. scripts/check.sh on autopilot/integration at 226c20e: ALL CHECKS PASSED; claims OK, client 255, findings 129 (autopilot tasks/review-rows-128-129/check-226c20e.log).

Limits

  • Three moves a → b, b → a, a → b inside 10 minutes mute the third, so the last alert names the endpoint the guardian left.
  • Some moves leave an endpoint that still answers: crossCheck finds it behind or missing recent transactions, or it serves another cluster. These log the same "is failing" line, and its error says why (B's backlog c43).
  • The wrong-cluster line (main.mjs:147) puts the entry's index past unit 60 when the host is 40 units or longer. So a second backup entry on such a host is muted the same way. No test reaches that line (B's backlog c44).
  • Not tested against a real Slack or Discord (localhost only).

🤖 Generated with Claude Code

shuv-amp and others added 2 commits October 19, 2026 20:53
An RPC failover logs `RPC endpoint <from> is failing; now using <to>`,
and alertFor keyed a line with no subject and no errorCause on its
message's first 60 units. From a 24-unit from-host on
(solana-mainnet.g.alchemy.com is 28), <to> starts past unit 60, so a
move a -> b and then a -> c within the 10-minute window was one kind and
the second was muted: the operator was not told the guardian now reads
from c (finding 129). The message also named hosts only, so two entries
on one host (FailoverRpc: "a URL listed twice does not name its entry")
read the same.

Fix: the failover line names both endpoints by entry, as the cluster
lines do (endpointName: `host (rpc)`, `host (rpcs[0])`), and in
fields, `endpointFrom` and `endpoint`, which about() counts as parts of
the subject, like a change, program or mint. The same move still repeats once a window. The
parts are matched bare, so the detail does not name them again when the
message says them; a message cut at MESSAGE_MAX names them at the end,
as for the other parts.

Tests:
- alert.test.mjs "a failover to another endpoint is news even when its
  host falls past the key's 60 units (finding 129)": a->b, a->b muted,
  a->c alerts with the detail unchanged, a->d (c's host, another entry)
  alerts, c->b alerts, and two moves to b from two entries on one
  49-unit host both alert (their names differ past unit 60).
- restart.test.mjs (finding 114 start failover): the real line reads
  `RPC endpoint <host> (rpc) is failing; now using <host> (rpcs[0])` and
  carries endpointFrom/endpoint = `<host> (rpc)`/`<host> (rpcs[0])`.
- tasks/failover-key/mutants3.log: 38dc10f's code with the new tests
  RED; 9/9 mutants KILLED (no endpoint part, no from part, either not
  bare; line without either field; endpoint = from; from = to; both by
  host); restored GREEN.
- scripts/check.sh: ALL CHECKS PASSED (tasks/failover-key/check.log,
  before the from field; check3.log on this commit, check-exit=0).
- docs/09:40 client tests 255; tools/claims.py OK, client 255.

Code review (high, three passes): fixed the entry naming, the from-entry
past unit 60, three comments and a doubled hostOf. Declined: keying
every line on its whole message with digits normalised (lines quoting a
signature or mint would then never repeat as one kind). Not fixed here,
backlog c42: moves a->b, b->a, a->b within the window mute the third, so
the last alert names the endpoint the guardian left; never muting a move
trades that for one alert per switch when two endpoints flap. Backlog
c43: a cross-check move (crossCheck, stale endpoint) logs the same "is
failing" line; its reason is logged on its own warn line
(main.mjs:1488).

Sibling search (main.mjs, every error/warn line with no errorCause): the
other lines whose distinguishing words sit past unit 60 carry a subject
field (changeId 735, ticketId 624, program 875, mint 340) or start with
what differs (1232, 1334). Not fixed here, backlog c44: the wrong-
cluster line (147) names its entry at unit 13, but one host of 40+ units
listed twice puts the entry's index past unit 60, and onCluster says it
once per endpoint, so the second is never heard; no test reaches line
147 today. stop.mjs's failover line sends no alerts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The commit before this one keys a failover's repeated alert on the
endpoints it moved from and to. Its line had no subject, so alertFor
keyed on its first 60 units, which held only the first 24 units of the
new endpoint less the first host's length: two moves from one endpoint
to hosts that begin alike were one kind and the second was muted, and
from a 24-unit first host on every move from it was. This gives it a
number.

docs/09 §4 gets row 129, and row 128's limit on the failover's key
points to it. README and docs/12 count 129 findings, 96–129 recorded
after the submission, and 129 among the guardian's fixed. No code
changes.

The row's "Not covered" names the a -> b, b -> a, a -> b case (the
third is muted, so the last alert names the endpoint the guardian
left), a move off an endpoint that answers (crossCheck's behind or
missing recent transactions, call()'s wrong cluster: the line says "is
failing", its error says why; backlog c43), and the
wrong-cluster line's like key from a 40-unit host (backlog c44).

Evidence:
- Units (node): "RPC endpoint " 13, " is failing; now using " 23,
  "solana-mainnet." 15, "solana-mainnet.g.alchemy.com" 28; "RPC
  endpoint " + a 40-unit host + " (rpcs[" is 60, so the index digit is
  unit 60 on. main.mjs:133 names the primary `host (rpc)`.
- The moves that are not failures: solana.mjs:495 ("RPC endpoint is
  behind"), :528 ("missing recent transactions"), :562 ("serves
  another cluster"); each goes through switchTo -> onSwitch, which
  logs main.mjs:141's line.
- python3 tools/claims.py: claims OK, findings 129, client 255.

My code review (high) of the record found the collision stated only
for a long first host, row 128 not pointing here, "Not fixed" where
every other row says "Not covered", the crossCheck move left out, the
example name `host (rpcs[0])` that misses the primary's `(rpc)` (also
in alert.mjs's comment, fixed in the commit before), the wrong-cluster
threshold left vague (and given as 36 in an earlier draft of the
commit before; 40 is right), and row 128's "160–300 ... never" (fixed
in row 128's commit). All fixed. A second pass found the 15-unit
example's own condition (a 9-unit first host) unstated, "has no
subject" in the present tense, and crossCheck's second move and
call()'s wrong-cluster move left out: fixed. It also asked to give the
wrong-cluster line its `endpoint` field here; declined: no test reaches
main.mjs:147, so that fix needs its own test and commit (backlog c44).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shuv-amp added a commit that referenced this pull request Oct 19, 2026
…es (#138-#147)

Batched squash of the reviewed, stacked PRs #138-#147 (one tree, identical to the reviewed stack's; each PR reviewed adversarially and green in CI on its own head):

- #138 Test the Vault's record checks with forged records (finding 120)
- #139 Record findings 117 to 124 in docs/09 section 4
- #140 Cut a long RPC error in an alert so the ids after it reach Discord (finding 125)
- #141 Pin G3's health bound at its edge with an exhaustive test
- #142 Key a failed action's repeated alert on its error's cause (finding 126)
- #143 Offer the sweep forged owner-checked accounts, swapped at every slot (finding 127)
- #144 Show an RPC's error in an alert whose line has a message (finding 128)
- #145 Key an RPC failover's alert on the endpoints it moved between (finding 129)
- #146 The sweep copies every account of the swept program, not only named ones
- #147 Key a backup's wrong-cluster alert on its endpoint entry (finding 130)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shuv-amp

Copy link
Copy Markdown
Owner Author

Included in main as d0177d5, the batched squash merge of #138-#147 via #147. main's tree there equals this stack's reviewed tree.

@shuv-amp shuv-amp closed this Oct 19, 2026
@shuv-amp
shuv-amp deleted the auto/B/failover-key-r2 branch October 19, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant