Repository navigation
Conversation
followConfigs (cloud/guardian/follow.mjs) ingested a watched Floor config as a replay (act=false) until the config was in `followed`, which happens only once a poll's walk and ingest succeed. So the line between replay and live was the first good poll, not the start of the process. Two cases read events after the start silently: - A restart whose replay failed (a refused walk, a transaction requireTransaction cannot get) stayed a replay on every later poll, and all that landed in the meantime was replayed once a poll got through. - A restart whose first poll succeeded still replayed what landed between the start and that poll. The first followFloor runs after the Vault startup: its replay (up to 50,000 signatures, no budget), the chain sync and the startup judgement. Replay sends nothing and logs no event (consider in main.mjs). A replaced or removed recovery key, a restore and its approvals are reported only through these events, so they were never said. A stop and an ejection showed only in the minute's config check (readiness "stopped", the guardian set), without who did it. The account scans still veto pending proposals, so the loss is the reports, not vetoes. The test added in 221b13f pins that a failed replay stays a replay; it did not ask what lands after the start. Fix: the line is the slot the process started at, on every poll. - main() reads it once, after the cluster check, with startSlot (follow.mjs): the highest confirmed slot that any of a FailoverRpc's endpoints gives, so one endpoint behind cannot put the line early. If none answers (an endpoint asked directly leaves at once when busy: `hurry`, 08b1822), or for a plain Rpc, the RPC itself is asked, and FailoverRpc.call waits for a busy one once. With nothing answering it is fatal (must), like the cluster check. followFloor passes the slot to followConfigs as `liveAfterSlot`. - The node lists signatures newest-first by slot, and the walk already relies on that (afterCursor, `until`). So one cut splits a poll: the first row in a slot at or before the start slot, and all listed after it, are replayed, then the newer rows are ingested live. A row with no slot (null over JSON-RPC) never ends the newer part, so one among the newest is acted on (twice at worst, never unsaid). That holds for a followed config too, so a node whose listing had not reached the start slot at the first poll cannot make older rows live at the next; a followed config that lists such rows logs a warning once they are read (a lagging node, or a start slot from an endpoint ahead of its chain), with the start slot and their slots. The warning alerts (alert.mjs URGENT_WARNINGS), as Floor's stop and ejection do since finding 76: with a late line, the events in those rows are never reported. Its text gives both causes, the count, the lowest and highest slot and the start slot, since the webhook shows only the text and the program. Slots, not block times: a block time against the host clock moves with clock skew, and block times need not grow. Without a start slot (only when no Floor config is watched) the old rule stands: resuming replays, followed acts. `ingest` is now called only with something to ingest: the empty calls did nothing in main.mjs, and two existing assertions that pinned an empty replay call now pin no call. Evidence (tasks/floor-replay-boundary/ in the autopilot directory): - New test "a Floor config resuming acts on what lands after the slot the process started at" (restart.test.mjs), start slot 4 unless said: a failed replay, then s4-s7 land (s4 in the start slot, s6 with an earlier block time); the next poll replays s4, s3 and acts on s7, s6, s5 in that order, and the config is followed. A clean first poll replays s3 and acts on s5, and logs nothing. All after the start (2): all live, no empty replay call. All at or before it (3): all replayed, no live call. A followed config listing s5, s4, s3: s4 and s3 replayed, s5 live, one warning naming 2 rows, start slot 4 and slots 3-4; the same with the replay failing: no warning. A null-slot row on top: live. One between s7 and s5: live with them. One below s4: replayed with s4 and s3 in one pass, the cursor at s4. - New test "the start slot is the highest confirmed slot any endpoint gives, and none is fatal": highest of four endpoints (one behind, one refusing), each asked for its confirmed slot, and the RPC's own call not made; a plain Rpc asked itself; every endpoint busy but the RPC's own call answering; nothing answering throws. - red-r9.log, the r9 test file: (a) on the parent's follow.mjs with only startSlot appended: the boundary test acts on nothing, and the empty-call pin fails (11 pass, 2 fail). (b) on the version before the sixth code review (branch auto/B/floor-replay-boundary-r7, kept): the warning has no start slot or slots (12 pass, 1 fail). red-r8.log: the previous test file on the version before the fifth (-r6): the null row below s4 makes it act on sn and stop with s4 unread. clean-restart-probe-r5.log: the clean-first-poll case alone on the parent's follow.mjs: [["P1",false,["s5","s3"]]], s5 never acted on. - mutants-r9.py / mutants-r9.log (full client suite each, on the r9 tests): M1-M16, M19-M21 and S1-S9 KILLED: the first-good-poll line, the start slot live, no slot as all live, no slot as all replay on a followed config, the line only while resuming, a null slot ending the newer part, all newer as none, an empty replay call, an empty live call, live as replay, replay as live, live before replay, the replay part from the front, no warning, a warning on a resuming config, the warning's row count, the warning before the replay is read, without the start slot, its slots newest first; startSlot lowest, no fallback to the RPC, the RPC asked always, a refusal not caught, none not fatal, processed commitment, first answer, endpoint refusals kept, the RPC's refusal kept. M17 and M18 (main.mjs does not pass or does not read the slot) SURVIVE: no test drives main() or followFloor, as for the rest of their wiring (221b13f); the spawn test in restart.test.mjs stops at the key file, before main(), and reaching followFloor would take a fake of the whole startup RPC surface. Baseline green; both files restored (cmp). Earlier rounds are kept beside it; mutants-r4-INVALID.log and mutants-r7-OVERLAP-INVALID.log are void and say why. - Review by A (CHANGES): the warning was level warn and matched no URGENT_WARNINGS pattern, so alertFor returned null and it went only to stdout. Now a pattern; the lagging-node case in restart.test.mjs passes the logged line itself to alertFor and expects a warning whose detail ends with the count, slots, start slot and program. A followed config whose replay part is out of slot order with a slot-less row last pins the slots as the lowest and highest of the rows that have one (it logged [null, 4]). Its lowest row sits in the middle, so taking the first and last rows instead fails too. mutants-r12.py, mut-r12-N1..N6.log, the whole client suite each on the final tests: the lowest slot as the last row's (N1), no numbers in the text (N2), the pattern removed (N3), the lowest as the highest (N4), the highest as the first row's (N5), the old wording (N6): all KILLED, 203 pass, 1 fail; files restored (cmp); green-r12.log 204/204. red-r9 and mutants-r9 (above) ran on the tests before this round. - Second review by A (CHANGES, low): its mutant R8 (the slots as the first and last rows', sorted) survived, since that case had its extremes at the ends. The case now has slot 1 between slots 3 and 4. r14/mutants.log, A's mut.py on the whole client suite: R1, R2, R4, R8, R9 KILLED; follow.mjs restored (cmp); r14/green.log 204/204. - Client suite 204/204 (202 on the parent, two new tests); docs/09:40 says 204 in both columns. - scripts/check.sh at this commit on 1646a1a: ALL CHECKS PASSED, client 204/204, claims OK (r14/check.log); check-each-commit.sh 1646a1a: green (r14/each-commit.log). Limits: the main.mjs wiring is not pinned (M17, M18). The split assumes the node's slot order, as the walk does; a listing out of slot order is split at its first row at or before the line. With a single endpoint that lags, or every endpoint lagging, the start slot is early, and what landed shortly before the start (while no guardian ran) is acted on as new: alerts for real events, and a veto for each proposal among them that is not approved (the account scans send those anyway while it is pending; for one already executed the veto fails, and how its retry ends was not checked here). An endpoint whose slot is ahead of its chain, or a backup on another cluster (checkCluster asks only the active endpoint), puts the line late, and it holds for the life of the process: until the watched chain reaches it, what a followed config lists is replayed. That is worse than the old rule, which ended at the first good poll. The followed config's warning shows it, and it alerts, at most once per 10 minutes per program (alertFor's window) (backlog c8: check every endpoint's cluster). A lagging node alerts the same way, though its rows landed before the start and a resuming replay would also replay them silently: the two cannot be told apart. Under a late line the first poll, which is a resuming replay, replays what landed after the start and below the line without a warning: only rows a followed config lists alert. A replay part that checkpoints some rows and then fails logs nothing, and the next poll's warning counts only what is left. A replay part still has no budget (main.mjs: `live ? ingestBudgetMs : Infinity`), now on a followed config too when its listing reaches back before the start (backlog c3p; a slot line makes budgeting the replay safe, since the rest is still a replay next poll). The replay part uses requireTransaction: one old transaction the RPC will not return blocks the poll (the stall alert fires; c7). A config with no cursor (a first start, a lost state file) still anchors at its newest signature at the first poll, so what lands between the start and that poll is not read at all (finding 71's "the account scans cover what came before"; c6). A row with no slot is checkpointed at slot 0, as before, which drops the next walk's minContextSlot guard; afterCursor drops such rows whenever the cursor has a slot. startSlot waits for its slowest endpoint once at startup (an endpoint's timeout is 10 s) and repeats crossCheck's ask-every-endpoint pattern rather than share a helper. Rows in the seconds around the slot read can fall either side: those landing before it are replayed, and some processed before the confirmed slot read but confirmed after it are acted on. The line is this process's start: a row the previous process listed live but did not finish (a transaction that kept failing, a crash) is replayed after a restart, as the old code did (backlog c9). Sibling search: the only other replay/live line is the Vault startup ingest (main.mjs, act=false). Its line is its own startup listing (the live loop reads on from `until`, the newest signature listed), and a failure there ends the process (main().catch(fatal)), so it never runs on as a replay. refreshPendingTickets and syncFromChain take `act` from their caller, and spendExecution's `live` and assess's `act` consume the flag; none ties it to a poll succeeding. The no-cursor anchor above is the one sibling found; it is left for its own change (c6). Code review (high), eight passes. - Pass 1, seven points. Taken: the slot from every endpoint; fatal when none answers (it was a warning and a silent fallback to the old rule); no empty replay call; the followFloor comment. Stated: the wiring, the no-cursor anchor. Not taken: "must() retries for minutes before the slot is read": must (main.mjs:118) ends the process on the first failure. - Pass 2, nine points. Taken: the line on every poll (a lagging listing made a followed config act on older rows); the RPC's own call in startSlot (all endpoints busy was fatal); slot-less rows; the doc header. Stated: a blocked replay, a cross-cluster backup, the shared helper, the startup wait. - Pass 3, five points. Taken: the RPC's own call only as a fallback (it asked the active endpoint twice); the cross-cluster Limit corrected (it said "the old behaviour"). Not taken: making `liveAfterSlot` required; without it the old rule is kept for the existing tests, and main.mjs always sets it when a Floor config is watched. Its other point (replay rows first could checkpoint past a live row in an out-of-order listing) led to runs in listing order, undone in pass 5. - Pass 4, seven points. Taken: a null slot is live (`null <= n` is true in JS, so it was replayed); the followed config's warning (a bad line was silent); `.catch`. Stated: the followed config's unbudgeted replay, the slot-0 checkpoint. Its per-run budget and cut-run points were about the runs, gone in pass 5. - Pass 5, four points, all taken: the runs (with a stop after a live run) dropped an older replay row for good in an out-of-order listing, left rows that the next poll took for a lagging node, and pinned a cut-replay path main.mjs never produces. Since the walk assumes slot order anyway, one cut, replay part then live part, replaces them. - Pass 6, five points. Taken: the warning only once the replay is read (a failing one repeated it each poll), with the start slot and slots; the Limits on rows around the slot read. Stated: rows a previous process left unfinished (c9; not a regression). Not taken: pinning the main.mjs wiring by spawning main.mjs (see the mutants above). - Pass 7, after A's review, eight points. Taken: the alert carries the count and slots; the slots skip slot-less rows; the alert.mjs comment had the line backwards; the evidence cited the r9 runs. Stated: a lagging node alerts; a failed partial replay is undercounted. Not taken: `notify: true` instead of a pattern (URGENT_WARNINGS is how a warn line alerts, and the test feeds it the logged line, so a reworded message fails it: N2/N3); a rate-limit test (alertFor's window and per-program key are alert.test.mjs's). - Pass 8, nine points. Taken: the text said the rows were from before the start, which is false with a late line (now "at or before the start slot", with both causes); the highest slot (it was the first row's); the slot test matches newerThan's (`!= null`); the alert.mjs comment now says a first poll's replay is silent; three commit message slips. Stated: a first resuming poll replays rows under a late line silently (below); the startSlot/crossCheck copy. Not taken: level error. The hidden events alert as warnings themselves (a key replaced or removed), so a warning matches them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shuv-amp
force-pushed
the
auto/A/harbor-masks
branch
from
October 18, 2026 20:05
1646a1a to
7c5ce6e
Compare
shuv-amp
force-pushed
the
auto/B/floor-replay-boundary-r11
branch
from
October 18, 2026 20:05
b6bf1d7 to
306bad8
Compare
shuv-amp
added a commit
that referenced
this pull request
Oct 19, 2026
… config checks (#120-#128) Batched squash of the reviewed, stacked PRs #120-#128 (one tree, identical to the reviewed stack's; each PR reviewed adversarially and green in CI on its own head): - #120 The sweep requires each program's own refusal code; pin the Vault caller checks (finding 112) - #121 Assert the program's own code at six refusals that took any error - #122 Fuzz elf_extent and snapshot_restorable against their rules in u128 - #123 Swap a vault's accounts whole for another protocol's in the sweep (finding 113) - #124 Act on Floor events that land after the guardian starts - #125 Use an RPC endpoint only once its genesis hash matches (finding 114) - #126 Read the emergency stop only from endpoints on its cluster - #127 Guardian config: refuse cluster lists, inherited names, non-object policy - #128 Guardian config: refuse unknown alert and liveness keys; whole numbers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Guardian (cloud/) and one test count in docs/09. No program change. One commit, 306bad8.
The guardian acts on Floor events that land after it starts.
followConfigs(cloud/guardian/follow.mjs) treated a watched Floor config as a replay (send nothing, log no event) until its first poll succeeded. So a restart whose replay failed stayed a replay on every later poll, and even a clean restart replayed what landed between the start and its first poll (that poll runs after the whole Vault startup). A replaced or removed recovery key, a restore and its approvals are reported only through these events, so they went unsaid. The account scans still veto pending proposals: the loss was the reports, not vetoes.Fix: the line is the slot the process started at.
main()reads it once after the cluster check (startSlot: the highest confirmed slot any endpoint gives; fatal if none answers). Each poll splits its listing there: rows at or before the start slot are replayed, newer rows are acted on. A followed config that still lists rows at or before the line logs a warning that alerts (a lagging node, or an endpoint ahead of its chain), with the count, the lowest and highest slot and the start slot.Proof
Autopilot tasks/floor-replay-boundary/:
scripts/check.shALL CHECKS PASSED andbin/check-each-commit.shgreen on 1646a1a (r14/).Limits
main.mjspassing and reading the slot is not pinned by a test (M17, M18 survive): no test drivesmain()orfollowFloor.The commit message lists eight code-review passes and what was taken, stated or not taken from each.
Hashes after the sync onto main (19 Oct)
The sync replayed the stack onto main bb55c15 (#114), whose tree is auto/B/failover-snapshot-r2's (8506759); #113's is auto/B/floor-follow-each's (51d20d3), both built on the main they merged into. The owner's #107 edited one sentence in README.md, docs/08 and docs/09 (the devnet job, and with it
ci, fails for Floor and Vault until finding 107 is deployed); and #99's merge changed README's devnet line and the guardians figure (3-8 -> 4-8 keys). Each commit's own message is unchanged. Every old/new pair's trees differ by exactly that: README.md, docs/08, docs/09, docs/assets/src/figures.py and the two PNGs, the same +/- lines for all 38 synced commits (autopilot tasks/sync-main-114/per-commit-tree.log and map.txt). The ids under What changed are the new ones; the runs and reviews quoted elsewhere were done at the old ones. Old = new: b6bf1d7 = 306bad8. On the synced chain:bin/check-each-commit.sh bb55c1541/41 green (the 38 synced commits, then the murmur3 exclusion fix, finding 106's citation and the full-ledger fuzz target), andscripts/check.shALL CHECKS PASSED at their tip (tasks/sync-main-114/each-commit.log, check-tip.log). Agent B reviewed the sync: REVIEWS/auto-A-sync-main-114.md.🤖 Generated with Claude Code