Skip to content

Act on Floor events that land after the guardian starts - #124

Closed
shuv-amp wants to merge 1 commit into
auto/A/harbor-masksfrom
auto/B/floor-replay-boundary-r11
Closed

shuv-amp wants to merge 1 commit into
auto/A/harbor-masksfrom
auto/B/floor-replay-boundary-r11

Conversation

@shuv-amp

@shuv-amp shuv-amp commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Opened by the Bulkhead autopilot agents: one reviewed unit of work. Every commit was reviewed adversarially by the other agent, with mutants, and passes tools/claims.py on its own.
Stacked on #123: merge that first; this PR is then rebased onto main.
The agents never merge. The owner reviews and merges.

What changed

Guardian (cloud/) and one test count in docs/09. No program change. One commit, 306bad8.

The guardian acts on Floor events that land after it starts. followConfigs (cloud/guardian/follow.mjs) treated a watched Floor config as a replay (send nothing, log no event) until its first poll succeeded. So a restart whose replay failed stayed a replay on every later poll, and even a clean restart replayed what landed between the start and its first poll (that poll runs after the whole Vault startup). A replaced or removed recovery key, a restore and its approvals are reported only through these events, so they went unsaid. The account scans still veto pending proposals: the loss was the reports, not vetoes.

Fix: the line is the slot the process started at. main() reads it once after the cluster check (startSlot: the highest confirmed slot any endpoint gives; fatal if none answers). Each poll splits its listing there: rows at or before the start slot are replayed, newer rows are acted on. A followed config that still lists rows at or before the line logs a warning that alerts (a lagging node, or an endpoint ahead of its chain), with the count, the lowest and highest slot and the start slot.

Proof

Autopilot tasks/floor-replay-boundary/:

  • Two new tests in cloud/test/restart.test.mjs (the boundary on resuming and followed configs, null-slot rows, the warning; the start slot from many endpoints, and none fatal). Red on the old code (red-r9.log, red-r8.log, clean-restart-probe-r5.log).
  • Mutants on the whole client suite: M1-M16, M19-M21, S1-S9 KILLED (mutants-r9.log); N1-N6 KILLED (mut-r12-*.log); A's R1, R2, R4, R8, R9 KILLED (r14/mutants.log).
  • Client suite 204/204; scripts/check.sh ALL CHECKS PASSED and bin/check-each-commit.sh green on 1646a1a (r14/).
  • Agent A reviewed it three times: CHANGES (the warning did not alert), CHANGES low (mutant R8 survived; a case with the lowest slot in the middle now kills it), then APPROVE (REVIEWS/auto-B-floor-replay-boundary-r11.md).

Limits

  • main.mjs passing and reading the slot is not pinned by a test (M17, M18 survive): no test drives main() or followFloor.
  • The split assumes the node lists in slot order, as the walk already does.
  • A lagging start slot makes what landed shortly before the start count as new (alerts, and vetoes for unapproved proposals). An endpoint ahead of its chain, or a backup on another cluster, puts the line late for the life of the process; the warning shows it. Checking every endpoint's cluster is the next unit (cluster-each-endpoint, finding 114).
  • A config with no cursor (first start, lost state file) still anchors at its newest signature, so rows between the start and the first poll are not read (backlog c6). The replay part has no time budget (c3p).

The commit message lists eight code-review passes and what was taken, stated or not taken from each.

Hashes after the sync onto main (19 Oct)

The sync replayed the stack onto main bb55c15 (#114), whose tree is auto/B/failover-snapshot-r2's (8506759); #113's is auto/B/floor-follow-each's (51d20d3), both built on the main they merged into. The owner's #107 edited one sentence in README.md, docs/08 and docs/09 (the devnet job, and with it ci, fails for Floor and Vault until finding 107 is deployed); and #99's merge changed README's devnet line and the guardians figure (3-8 -> 4-8 keys). Each commit's own message is unchanged. Every old/new pair's trees differ by exactly that: README.md, docs/08, docs/09, docs/assets/src/figures.py and the two PNGs, the same +/- lines for all 38 synced commits (autopilot tasks/sync-main-114/per-commit-tree.log and map.txt). The ids under What changed are the new ones; the runs and reviews quoted elsewhere were done at the old ones. Old = new: b6bf1d7 = 306bad8. On the synced chain: bin/check-each-commit.sh bb55c15 41/41 green (the 38 synced commits, then the murmur3 exclusion fix, finding 106's citation and the full-ledger fuzz target), and scripts/check.sh ALL CHECKS PASSED at their tip (tasks/sync-main-114/each-commit.log, check-tip.log). Agent B reviewed the sync: REVIEWS/auto-A-sync-main-114.md.

🤖 Generated with Claude Code

followConfigs (cloud/guardian/follow.mjs) ingested a watched Floor
config as a replay (act=false) until the config was in `followed`, which
happens only once a poll's walk and ingest succeed. So the line between
replay and live was the first good poll, not the start of the process.
Two cases read events after the start silently:
- A restart whose replay failed (a refused walk, a transaction
  requireTransaction cannot get) stayed a replay on every later poll,
  and all that landed in the meantime was replayed once a poll got
  through.
- A restart whose first poll succeeded still replayed what landed
  between the start and that poll. The first followFloor runs after the
  Vault startup: its replay (up to 50,000 signatures, no budget), the
  chain sync and the startup judgement.
Replay sends nothing and logs no event (consider in main.mjs). A
replaced or removed recovery key, a restore and its approvals are
reported only through these events, so they were never said. A stop
and an ejection showed only in the minute's config check (readiness
"stopped", the guardian set), without who did it. The account scans
still veto pending proposals, so the loss is the reports, not vetoes.
The test added in 221b13f pins that a failed replay stays a replay; it
did not ask what lands after the start.

Fix: the line is the slot the process started at, on every poll.
- main() reads it once, after the cluster check, with startSlot
  (follow.mjs): the highest confirmed slot that any of a FailoverRpc's
  endpoints gives, so one endpoint behind cannot put the line early. If
  none answers (an endpoint asked directly leaves at once when busy:
  `hurry`, 08b1822), or for a plain Rpc, the RPC itself is asked, and
  FailoverRpc.call waits for a busy one once. With nothing answering it
  is fatal (must), like the cluster check. followFloor passes the slot
  to followConfigs as `liveAfterSlot`.
- The node lists signatures newest-first by slot, and the walk already
  relies on that (afterCursor, `until`). So one cut splits a poll: the
  first row in a slot at or before the start slot, and all listed after
  it, are replayed, then the newer rows are ingested live. A row with no
  slot (null over JSON-RPC) never ends the newer part, so one among the
  newest is acted on (twice at worst, never unsaid). That holds for a
  followed config too, so a node whose listing had not reached the
  start slot at the first poll cannot make older rows live at the next;
  a followed config that lists such rows logs a warning once they are
  read (a lagging node, or a start slot from an endpoint ahead of its
  chain), with the start slot and their slots. The warning alerts
  (alert.mjs URGENT_WARNINGS), as Floor's stop and ejection do since
  finding 76: with a late line, the events in those rows are never
  reported. Its text gives both causes, the count, the lowest and
  highest slot and the start slot, since the webhook shows only the
  text and the program.
Slots, not block times: a block time against the host clock moves with
clock skew, and block times need not grow. Without a start slot (only
when no Floor config is watched) the old rule stands: resuming replays,
followed acts. `ingest` is now called only with something to ingest:
the empty calls did nothing in main.mjs, and two existing assertions
that pinned an empty replay call now pin no call.

Evidence (tasks/floor-replay-boundary/ in the autopilot directory):
- New test "a Floor config resuming acts on what lands after the slot
  the process started at" (restart.test.mjs), start slot 4 unless said:
  a failed replay, then s4-s7 land (s4 in the start slot, s6 with an
  earlier block time); the next poll replays s4, s3 and acts on s7, s6,
  s5 in that order, and the config is followed. A clean first poll
  replays s3 and acts on s5, and logs nothing. All after the start (2):
  all live, no empty replay call. All at or before it (3): all replayed,
  no live call. A followed config listing s5, s4, s3: s4 and s3
  replayed, s5 live, one warning naming 2 rows, start slot 4 and slots
  3-4; the same with the replay failing: no warning. A null-slot row on
  top: live. One between s7 and s5: live with them. One below s4:
  replayed with s4 and s3 in one pass, the cursor at s4.
- New test "the start slot is the highest confirmed slot any endpoint
  gives, and none is fatal": highest of four endpoints (one behind, one
  refusing), each asked for its confirmed slot, and the RPC's own call
  not made; a plain Rpc asked itself; every endpoint busy but the RPC's
  own call answering; nothing answering throws.
- red-r9.log, the r9 test file: (a) on the parent's follow.mjs with
  only startSlot appended: the boundary test acts on nothing, and the
  empty-call pin fails (11 pass, 2 fail). (b) on the version before the
  sixth code review (branch auto/B/floor-replay-boundary-r7, kept): the
  warning has no start slot or slots (12 pass, 1 fail). red-r8.log: the
  previous test file on the version before the fifth (-r6): the null
  row below s4 makes it act on sn and stop with s4 unread.
  clean-restart-probe-r5.log: the clean-first-poll case alone on the
  parent's follow.mjs: [["P1",false,["s5","s3"]]], s5 never acted on.
- mutants-r9.py / mutants-r9.log (full client suite each, on the r9
  tests): M1-M16, M19-M21 and S1-S9 KILLED: the first-good-poll line,
  the start slot live, no slot as all live, no slot as all replay on a
  followed config, the line only while resuming, a null slot ending the
  newer part, all newer as none, an empty replay call, an empty live
  call, live as replay, replay as live, live before replay, the replay
  part from the front, no warning, a warning on a resuming config, the
  warning's row count, the warning before the replay is read, without
  the start slot, its slots newest first; startSlot lowest, no fallback
  to the RPC, the RPC asked always, a refusal not caught, none not
  fatal, processed commitment, first answer, endpoint refusals kept, the
  RPC's refusal kept. M17 and M18 (main.mjs does not pass or does not
  read the slot) SURVIVE: no test drives main() or followFloor, as for
  the rest of their wiring (221b13f); the spawn test in restart.test.mjs
  stops at the key file, before main(), and reaching followFloor would
  take a fake of the whole startup RPC surface. Baseline green; both
  files restored (cmp). Earlier rounds are kept beside it;
  mutants-r4-INVALID.log and mutants-r7-OVERLAP-INVALID.log are void and
  say why.
- Review by A (CHANGES): the warning was level warn and matched no
  URGENT_WARNINGS pattern, so alertFor returned null and it went only
  to stdout. Now a pattern; the lagging-node case in restart.test.mjs
  passes the logged line itself to alertFor and expects a warning
  whose detail ends with the count, slots, start slot and program.
  A followed config whose replay part is out of slot order with a
  slot-less row last pins the slots as the lowest and highest of the
  rows that have one (it logged [null, 4]). Its lowest row sits in the
  middle, so taking the first and last rows instead fails too.
  mutants-r12.py, mut-r12-N1..N6.log, the whole client suite each on
  the final tests: the lowest slot as the last row's (N1), no numbers
  in the text (N2), the pattern removed (N3), the lowest as the
  highest (N4), the highest as the first row's (N5), the old wording
  (N6): all KILLED, 203 pass, 1 fail; files restored (cmp);
  green-r12.log 204/204. red-r9 and mutants-r9 (above) ran on the
  tests before this round.
- Second review by A (CHANGES, low): its mutant R8 (the slots as the
  first and last rows', sorted) survived, since that case had its
  extremes at the ends. The case now has slot 1 between slots 3 and 4.
  r14/mutants.log, A's mut.py on the whole client suite: R1, R2, R4,
  R8, R9 KILLED; follow.mjs restored (cmp); r14/green.log 204/204.
- Client suite 204/204 (202 on the parent, two new tests); docs/09:40
  says 204 in both columns.
- scripts/check.sh at this commit on 1646a1a: ALL CHECKS PASSED,
  client 204/204, claims OK (r14/check.log); check-each-commit.sh
  1646a1a: green (r14/each-commit.log).

Limits: the main.mjs wiring is not pinned (M17, M18). The split
assumes the node's slot order, as the walk does; a listing out of slot
order is split at its first row at or before the line. With a single
endpoint that lags, or every endpoint lagging, the start slot is early,
and what landed shortly before the start (while no guardian ran) is
acted on as new: alerts for real events, and a veto for each proposal
among them that is not approved (the account scans send those anyway
while it is pending; for one already executed the veto fails, and how
its retry ends was not checked here). An endpoint whose slot is ahead
of its chain, or a backup on another cluster (checkCluster asks only
the active endpoint), puts the line late, and it holds for the life of
the process: until the watched chain reaches it, what a followed
config lists is replayed. That is worse than the old rule, which ended
at the first good poll. The followed config's warning shows it, and
it alerts, at most once per 10 minutes per program (alertFor's
window) (backlog c8: check every endpoint's cluster). A lagging node
alerts the same way, though its rows landed before the start and a
resuming replay would also replay them silently: the two cannot be
told apart. Under a late line the first poll, which is a resuming
replay, replays what landed after the start and below the line
without a warning: only rows a followed config lists alert. A replay
part that checkpoints some rows and then fails logs nothing, and the
next poll's warning counts only what is left. A replay part still has
no budget (main.mjs: `live ? ingestBudgetMs : Infinity`), now on a
followed config too when its listing reaches back before the start
(backlog c3p; a slot line makes budgeting the replay safe, since the
rest is still a replay next poll). The replay part uses
requireTransaction: one old transaction the RPC will not return blocks
the poll (the stall alert fires; c7). A config with no cursor (a first
start, a lost state file) still anchors at its newest signature at the
first poll, so what lands between the start and that poll is not read
at all (finding 71's "the account scans cover what came before"; c6).
A row with no slot is checkpointed at slot 0, as before, which drops
the next walk's minContextSlot guard; afterCursor drops such rows
whenever the cursor has a slot. startSlot waits for its slowest
endpoint once at startup (an endpoint's timeout is 10 s) and repeats
crossCheck's ask-every-endpoint pattern rather than share a helper.
Rows in the seconds around the slot read can fall either side: those
landing before it are replayed, and some processed before the
confirmed slot read but confirmed after it are acted on. The line is
this process's start: a row the previous process listed live but did
not finish (a transaction that kept failing, a crash) is replayed after
a restart, as the old code did (backlog c9).

Sibling search: the only other replay/live line is the Vault startup
ingest (main.mjs, act=false). Its line is its own startup listing (the
live loop reads on from `until`, the newest signature listed), and a
failure there ends the process (main().catch(fatal)), so it never runs
on as a replay. refreshPendingTickets and syncFromChain take `act` from
their caller, and spendExecution's `live` and assess's `act` consume the
flag; none ties it to a poll succeeding. The no-cursor anchor above is
the one sibling found; it is left for its own change (c6).

Code review (high), eight passes.
- Pass 1, seven points. Taken: the slot from every endpoint; fatal when
  none answers (it was a warning and a silent fallback to the old
  rule); no empty replay call; the followFloor comment. Stated: the
  wiring, the no-cursor anchor. Not taken: "must() retries for minutes
  before the slot is read": must (main.mjs:118) ends the process on the
  first failure.
- Pass 2, nine points. Taken: the line on every poll (a lagging listing
  made a followed config act on older rows); the RPC's own call in
  startSlot (all endpoints busy was fatal); slot-less rows; the doc
  header. Stated: a blocked replay, a cross-cluster backup, the shared
  helper, the startup wait.
- Pass 3, five points. Taken: the RPC's own call only as a fallback (it
  asked the active endpoint twice); the cross-cluster Limit corrected
  (it said "the old behaviour"). Not taken: making `liveAfterSlot`
  required; without it the old rule is kept for the existing tests, and
  main.mjs always sets it when a Floor config is watched. Its other
  point (replay rows first could checkpoint past a live row in an
  out-of-order listing) led to runs in listing order, undone in pass 5.
- Pass 4, seven points. Taken: a null slot is live (`null <= n` is true
  in JS, so it was replayed); the followed config's warning (a bad line
  was silent); `.catch`. Stated: the followed config's unbudgeted
  replay, the slot-0 checkpoint. Its per-run budget and cut-run points
  were about the runs, gone in pass 5.
- Pass 5, four points, all taken: the runs (with a stop after a live
  run) dropped an older replay row for good in an out-of-order listing,
  left rows that the next poll took for a lagging node, and pinned a
  cut-replay path main.mjs never produces. Since the walk assumes slot
  order anyway, one cut, replay part then live part, replaces them.
- Pass 6, five points. Taken: the warning only once the replay is read
  (a failing one repeated it each poll), with the start slot and slots;
  the Limits on rows around the slot read. Stated: rows a previous
  process left unfinished (c9; not a regression). Not taken: pinning
  the main.mjs wiring by spawning main.mjs (see the mutants above).
- Pass 7, after A's review, eight points. Taken: the alert carries
  the count and slots; the slots skip slot-less rows; the alert.mjs
  comment had the line backwards; the evidence cited the r9 runs.
  Stated: a lagging node alerts; a failed partial replay is
  undercounted. Not taken: `notify: true` instead of a pattern
  (URGENT_WARNINGS is how a warn line alerts, and the test feeds it the
  logged line, so a reworded message fails it: N2/N3); a rate-limit
  test (alertFor's window and per-program key are alert.test.mjs's).
- Pass 8, nine points. Taken: the text said the rows were from before
  the start, which is false with a late line (now "at or before the
  start slot", with both causes); the highest slot (it was the first
  row's); the slot test matches newerThan's (`!= null`); the alert.mjs
  comment now says a first poll's replay is silent; three commit
  message slips. Stated: a first resuming poll replays rows under a
  late line silently (below); the startSlot/crossCheck copy. Not
  taken: level error. The hidden events alert as warnings themselves
  (a key replaced or removed), so a warning matches them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shuv-amp
shuv-amp force-pushed the auto/A/harbor-masks branch from 1646a1a to 7c5ce6e Compare October 18, 2026 20:05
@shuv-amp
shuv-amp force-pushed the auto/B/floor-replay-boundary-r11 branch from b6bf1d7 to 306bad8 Compare October 18, 2026 20:05
shuv-amp added a commit that referenced this pull request Oct 19, 2026
… config checks (#120-#128)

Batched squash of the reviewed, stacked PRs #120-#128 (one tree, identical to the reviewed stack's; each PR reviewed adversarially and green in CI on its own head):

- #120 The sweep requires each program's own refusal code; pin the Vault caller checks (finding 112)
- #121 Assert the program's own code at six refusals that took any error
- #122 Fuzz elf_extent and snapshot_restorable against their rules in u128
- #123 Swap a vault's accounts whole for another protocol's in the sweep (finding 113)
- #124 Act on Floor events that land after the guardian starts
- #125 Use an RPC endpoint only once its genesis hash matches (finding 114)
- #126 Read the emergency stop only from endpoints on its cluster
- #127 Guardian config: refuse cluster lists, inherited names, non-object policy
- #128 Guardian config: refuse unknown alert and liveness keys; whole numbers

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@shuv-amp

Copy link
Copy Markdown
Owner Author

Included in main as 982e781, the batched squash merge of #120-#128 via #128. main's tree there equals this stack's reviewed tree.

@shuv-amp shuv-amp closed this Oct 19, 2026
@shuv-amp
shuv-amp deleted the auto/B/floor-replay-boundary-r11 branch October 19, 2026 17:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant