Skip to content

Write log lines and webhook bodies as well-formed JSON - #137

Merged
shuv-amp merged 9 commits into
mainfrom
auto/B/json-well-formed
Oct 19, 2026
Merged

shuv-amp merged 9 commits into
mainfrom
auto/B/json-well-formed

Conversation

@shuv-amp

Copy link
Copy Markdown
Owner

Opened by the Bulkhead autopilot agents: one reviewed unit of work. Every commit was reviewed adversarially by the other agent, with mutants, and passes tools/claims.py on its own.
Stacked on #136: merge that first; this PR is then rebased onto main.
The agents never merge. The owner reviews and merges.

What changed

The JSON writer for log lines and webhook bodies (cloud/guardian/alert.mjs, main.mjs, stop.mjs), their tests and the client test count in docs/09 (247 -> 250). No program change. One commit, 18b2e53. Finding 124.

Log lines and webhook bodies are now well-formed JSON. The guardian and the stop cut an RPC's error text by UTF-16 units (.slice(0, 160), .slice(0, 200)). When the cut falls inside a surrogate pair, the line keeps the first half alone, and JSON.stringify writes it as "\ud83d". Strict parsers refuse the whole text: serde_json 1.0.151 answers "unexpected end of hex escape" for {"detail":"ab\ud83d"}. So a log shipper or JSON webhook receiver written that way drops that line or alert, and the alert is retried and then lost.

Fix at the writer, not at each cut: jsonText (alert.mjs) makes every string well-formed (U+FFFD for a half pair) and writes a bigint as its digits, as main.mjs's replacer did. It writes the guardian's log lines, the undelivered-alert line, the state-save line, the stop's log lines and every webhook body. The Slack and Discord bodies were already well-formed (finding 123); the JSON body was not.

Proof

Autopilot tasks/json-well-formed/ (runs taken before the rebase onto 804e297, at earlier ids of this commit):

  • New tests in alert.test.mjs, stop.test.mjs and restart.test.mjs. The commit states each fails at the parent: alert 0/1 pass, stop and restart 0/2.
  • mutants.log: 5 mutants (jsonText without toWellFormed, without the bigint branch, the body, main.mjs's log and stop.mjs's log via JSON.stringify), 5 KILLED, restored green.
  • check.log: scripts/check.sh ALL CHECKS PASSED. each-commit.log: green.
  • Agent A reviewed it with its own mutants (tasks/review-json-well-formed/): A1, A2, A5 KILLED. A3 and A4 (the undelivered-alert and state-save lines via JSON.stringify) and A6 (strings directly in an array skipped) SURVIVED; A judged them stated or equivalent for today's callers. APPROVE (REVIEWS/auto-B-json-well-formed.md). A also asked for a unit recording findings 117-124 in docs/09; that is the record-117-124 unit higher in this stack.
  • After the rebase onto 804e297 A checked the delta: only the docs/09 count and the count line differ; bin/check-each-commit.sh 804e297 all 7 green (tasks/review-rebase-35/). APPROVE.
  • At integration (19 Oct): bin/integrate.sh ran bin/check-each-commit.sh green on this unit's commit (autopilot tasks/integrate-it12/6.log), and scripts/check.sh at the top of this batch, fb44f95, which contains this unit unchanged: ALL CHECKS PASSED (tasks/integrate-it12/check-fb44f95.log).

Limits

  • The undelivered-alert and state-save lines in main.mjs have no test of their own; they use the same jsonText.
  • A string directly inside an array is not tested (A's N1). No log line today carries an array.
  • e2e/attack.mjs's demo log cuts e.message the same way and is left as is (a demo, not run by operators).

🤖 Generated with Claude Code

@shuv-amp
shuv-amp changed the base branch from auto/B/alert-text-escape-r3 to main October 19, 2026 17:45
shuv-amp and others added 9 commits October 19, 2026 23:30
Row 106 said its 16 Oct re-run was on tree 36187b4, "the tree of
86b777b (PR #100's head)". PR #100 was pushed again before the owner
merged it: refs/pull/100/head is now 7ded086, 86b777b is not an
ancestor of it, and no ref on origin (branches or refs/pull/*/head)
contains 86b777b or a commit with tree 36187b4. The citation named
something nobody can fetch.

What the re-run used is still on main. git diff 86b777b 04ad61e --
programs crates touches one file, programs/tests/tests/vault.rs, 2
lines: a doc comment of a_guardian_silent_past_its_interval_can_no_
longer_act rewrapped. So the row now cites main at 04ad61e (PR #100's
squash) and says that one difference.

Sibling search: every 7-12 hex token in README.md and docs/*.md (47)
resolved with git cat-file; each commit checked with for-each-ref
--contains against refs/remotes/origin and refs/pull/*/head. Only
86b777b was unpublished; 36187b4 was the only tree id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The ledger target had two gaps. It mapped every mint byte to one of 4
mints, so its ledgers never filled (MAX_SLOTS = 8) and backed_amount's
full-ledger refusal (ledger.rs:237) was never reached; docs/09 listed
this. And it never checked a grant against the rule: it asserted
backed <= amount, that a re-ask grants the same, and that debit_backed
accepts it, but debit_backed re-runs backed_amount and only a missing
slot could make it fail, so no unsound grant could fail the target.

The target now draws from MAX_SLOTS + 1 mints, each with its own price,
keeps the first MAX_SLOTS distinct nonzero positions (it took the first 8
raw tuples before), and checks every grant against sound_after, restated
from g3_backed_amount_is_sound_and_maximal (tests/properties.rs): the
grant is sound, and one unit more and the whole request are not.
from_positions can no longer refuse the ledger it builds, so it is
expected to succeed.

Its prices keep hi >= 1, as conservative() does (price > 0, hi >=
max(price, ema)). With hi = 0 a debt is worth 0, and backed_amount's
no-collateral shortcut (ledger.rs:254, "exactly what the search below
returns") grants 0 where the rule allows the whole amount: both 120 s
runs of the first version found it (tasks/fuzz-full-ledger/artifacts-
hi0: one debt in the asked mint, priced 0). No price the Vault reads is
0, and liability_bps >= 10,000 (genesis.rs:36, classify.rs:83), so a
debt is worth at least 1 on chain; the shortcut is exact there.

sound_after returns c > 0 when the health sum overflows, where the core
fails closed; not reachable: value_v caps a value at u128::MAX / BPS
(a 2e38 collateral counts 0, f1-overflow.log), and 9 of those fit i128.

Evidence (autopilot tasks/fuzz-full-ledger/, r6-mutants.sh/.log at this
head), each mutant 3 rounds of 60 s from a fresh corpus, and cargo test
-p bulkhead-core:
- M1 full-ledger refusal dropped (:237 `if false &&`): fuzz 3/2/2 s.
  The old target survived all three rounds (14.1M-14.8M runs each).
- M2 full at > MAX_SLOTS (:225): 5/8/3 s.
- M3 full at >= MAX_SLOTS - 1, M4 `!present` dropped (refuse too much):
  "maximal" in 4/2/1 and 3/2/2 s.
- M5 health bound t >= -1 (:246): "sound" in 2/2/2 s. No core test
  fails; check.sh's Python differential does (21/17/16 mismatches at
  seeds 11/12/13, 50,000 cases; m5-pyrust.log).
- M6 unknown health counted healthy (:242), M7 unallowlisted paid past
  its balance, M8 never paid (:231), M9 skip valid(amount) (:259): each
  fails within 2 s in all rounds.
- The whole-request check alone (the others disabled) kills M9 in
  2/1/1 s (allofit-M9-r*.log).
- Core tests catch M1-M4 and M6-M9 too; M5 is a first catch for cargo
  test only, and the differential has it.
- Real core: SECONDS_PER=300 scripts/fuzz.sh ledger, 45.6M runs, no
  crash.

The input layout changed, so saved ledger corpora (local and CI's
fuzz-corpus cache) decode into other values; libFuzzer reloads them as
plain inputs.

Sibling: docs/09 lists one other refusal no target asks for, the closed
limiter lane (capacity at least 1); left as is. The shortcut comment's
unstated hi >= 1 premise is a backlog item, not changed here (core
line numbers).
No program or core source change; sBPF hashes unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
crossCheck noted `active`, then awaited every endpoint's signature
list. A call() that failed over in that wait moved `active`, and the
check went on to switch away from the new endpoint on the old one's
list. A signature first seen missing on the old primary also kept its
clock across such a move, so the new primary was switched off at its
first check instead of after SIGNATURE_LAG_SWITCH_MS.

Root cause: the `missing` map recorded which backup had a signature
but not which endpoint lacked it, and the switch did not check that
the endpoint the lists judged was still active.

Fix: `missing` is one map per endpoint, filled from the lists of the
endpoint that was active when they were asked, so an endpoint call()
moved to gets its own 30 s. The switch happens only if that endpoint
is still active. A switch by crossCheck itself still clears every
map, as the single map was cleared before (`restartWaits`, which
also makes the maps lazily, so they follow `endpoints`). The
crossCheck comment says all this.

Two earlier versions were replaced after code review: clearing the
map on any move (call() moving e0 -> e1 -> e0 between checks would
restart the wait every time and keep the guardian on lagging
endpoints, where the old code switched), and dropping the whole round
when `switches` moved during it (e0 -> e1 -> e0 in flight, and the
lists still judge e0).

Evidence (tasks/crosscheck-switch/ in the autopilot dir):
- Nine tests in delivery.test.mjs against stub endpoints.
  r6/red-old.log: on the old code (0b233e9) three of 66 fail: the
  in-flight test ("no verdict on lists from the old primary"), the
  moved-to test ("e2 not
  judged on e0's wait") and the filed-under test ("e2's wait started
  at 50 s"). The other five pass on the old code and pin what it got
  right: the flap test and the moved-back test fail on the two
  earlier versions (M5, M3); the two clear tests pin the old clears
  of every map, which had no test (mutants-r2-partial.log: three such
  mutants survived before them); the left-in-flight test (from A's
  review) pins that lists asked of an endpoint call() left while they
  were in flight still record what it lacks, from every other
  endpoint, the one now active included: it runs once with S listed
  by e2 (now active) and once by e1. The ninth (A's, from the r3
  review) pins the other half: lists asked of an endpoint call() left
  in flight do not prune the wait of the one now active.
- r6/mutants13.log: 13/13 KILLED, none void, 1-9 of 66 fail (switch
  guard off/inverted, whole round dropped on any move, one shared map,
  clear on any move, evidence filed under the endpoint active after
  the await; on a slot switch: keep all maps, clear only the left,
  only the joined, or both; on a signature switch: keep all, clear
  only the left, or left and joined).
- r6/mutants-loop.log: A's X3 (record nothing when `active` moved in
  flight), X4 (backup filter reads this.active) and Y1 (when moved,
  record only from the endpoint now active) each fail the
  left-in-flight test alone; r4/mutants.log: X3 and X4 survive
  without it. r6/mutants-committed.log: Y1 survives the test with
  only e2 listing S, hence the second case.
- r7/ (on 859ed7e, after the ninth test): mutants.log and
  mutants-rest.log, M1-M13, X1-X4, X6-X8, Y1, Z1, Z2, Z4 KILLED, 1-14
  of 67 fail. A's Z4 (the prune loop reads this.missing[this.active])
  survived all 66 earlier tests (tasks/review-crosscheck-switch-r3/
  z4.log) and fails the ninth alone now. X5 survives (equivalent, A's
  r2 review), and so does Z3 (A's first, wrong form of Z4).
- node --test cloud/test: 237/237 (docs/09 client cell, 228 + 9).

Limits (backlog c16, c17, c18): the switch goes to the backup that
listed the signature when it was first seen missing, even if that
backup is down now. The old code did that too, but kept fewer such
entries: one became void when that backup was active and listed the
signature, and an endpoint's entries now outlive call()'s moves. An
inactive endpoint's wait keeps ageing (to the 600 s expiry), so one
call() returns to after a long absence is left at its first check.
An endpoint call() moves to is judged from the first check that finds
it active, so a lagging one is left a check later than before: up to
about two minutes at the default 60 s freshnessCheckS (before, it was
left at once, on the old primary's wait).

Sibling search: call() already dates its evidence with `switches`
(finding 84 family). crossCheck's slot step reads `active` after the
slot await and compares that same endpoint's slot, with no await
before its switch, so a move there is judged on the right endpoint.
No other reader of `missing` (git grep in cloud/).

Finding 119 reserved; its row waits on rows 114-118.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An emergency stop whose primary `rpc` does not parse as a URL (a typo
such as "http//host") gave up, though its backup was good. fetch
refuses the primary, so FailoverRpc moves to the backup and calls the
stop's onSwitch, which built its log line with `new URL(from).host`.
That threw inside call()'s try: the backup's answer was taken for its
failure, every read failed the same way, and the stop logged "gave up:
Floor's config could not be read within the time limit" with "Invalid
URL", exit 1 (finding 121).

Root cause: the stop kept its own URL parsing for logs, while the
guardian's copy (main.mjs, hostOf) had long caught the error; the two
drifted. Fix: one hostOf, exported from cloud/lib/solana.mjs and used
by both. It also gives "(endpoint)" for a URL with no host
("localhost:8899" parses with "localhost:" as its scheme), which
gave an empty name. The stop's switch line now names the config
entry, as its cluster lines do ("RPC (endpoint) (rpc) left; now using
host (rpcs[0])"); onSwitch is passed both indexes for that, since a
URL listed twice cannot name its entry. Why FailoverRpc let a hook's
error discard an answer is the next commit.

Evidence (tasks/switch-outside-try/; r2/red1.log, r3/):
- Test "a stop whose primary URL does not parse still stops through a
  backup" (stop.test.mjs). With the parent's stop.mjs, main.mjs and
  solana.mjs it fails, the stop "gave up" with "Invalid URL" (red1.log;
  r6/red-parent.log on this test as committed). Now exit 0, "stopped",
  and "RPC (endpoint) (rpc) left; now using host (rpcs[1])": the bad URL
  is listed twice, so the stop skips rpcs[0] and the line must name
  where it went.
- Tests "onSwitch is told each endpoint's index, a URL listed twice
  too" and "hostOf names a host, and only a host, for logs"
  (delivery.test.mjs).
- r3/mutants1.py (mutants1.log; re-run on this test: r6/mutants1.log,
  same), none void: S1 the switch line parses
  URLs itself, S2 hostOf rethrows, S3 the line names hosts only, S4
  hostOf gives an empty host, S5 switchTo gives no indexes: KILLED. S6
  the stop maps URLs back to indexes with indexOf SURVIVED, as
  expected: the stop never leaves a URL listed twice, so the stop's
  use of the indexes is covered only jointly with the lib test that
  they are passed (the test's comment says so). Y4 (A's review: the
  new entry named endpointName(fromAt + 1)) survived the first version
  of the test, with one backup; KILLED now (r6/mut-f1.log).
- node --test cloud/test/*.test.mjs: 240/240. claims OK, client 240.
  (The red and mutant logs were taken on the parent's earlier base,
  5a7c6b2, with 239 tests; A's ninth freshness test came in since.)

Sibling search: `git grep "new URL(" cloud` outside tests: the two
hostOf copies (now one), and status/snapshot.mjs:22, which builds an
error message only after a read of that same URL has answered, so it
parses there. cloud/status/index.html:248 has its own helper that
falls back to the raw URL; a URL it refuses would not have fetched
(backlog c19). Not done here: the stop does not refuse an unparseable
URL at start, as checkConfig does for the guardian; with good backups
it can still stop, so that is a choice (backlog c21).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FailoverRpc called its report hooks (onSwitch, onCluster) where their
errors were read as the endpoint's. onSwitch ran inside call()'s try:
a report that threw made the endpoint that had just answered look
failed, its answer was dropped, and the request, a sendTransaction
too, went on to the next endpoint while the dead one stayed active.
That is how the stop's log line (previous commit) turned one bad URL
into a stop that read nothing. onCluster ran in askGenesis after the
cluster was recorded, so a throw there passed over an endpoint on the
right cluster. In crossCheck, a throwing onSwitch rejected the check
before anything moved, so the guardian's loop (main.mjs, "rpc
freshness") saw a failed step instead of the switch.

Fix: both hooks go through report(), which passes over a hook's error
and writes one line to stderr ("FailoverRpc: onSwitch threw: ...", at
most 160 characters of the message, or of the value thrown when the
message is empty), so the failure still leaves a trace. The trace is
guarded too: String() of some thrown values (an object with no
prototype) throws. A hook for that failure (onHookError) was weighed and
not added: the hook that failed is the log. The shipped hooks no longer
throw (hostOf, previous commit); this keeps call() and crossCheck right
for any hook. Hooks are synchronous; a rejected promise from one is not
caught (none is async). Throwing the hook's error to the caller was
tried first (branch auto/B/switch-outside-try-r1) and dropped: it still
loses an accepted transaction's signature, and signer.mjs's send() would
then report a freeze that landed as failed.

Evidence (tasks/switch-outside-try/r4/):
- Four tests (delivery.test.mjs): a throwing onSwitch neither drops
  the answer nor sends the request on (three endpoints, the first
  down; also checks the stderr line); crossCheck's switch with a
  throwing onSwitch is made and returned to its caller (slot and
  signature causes); a throwing onCluster does not pass over an
  endpoint on the right cluster; a hook whose error cannot be printed,
  or is long, is passed over too (an object with no prototype, a
  Symbol, an Error with no message, a 161-character message). With
  the parent's solana.mjs: 4/243 fail (red2.log).
- mutants2.py, 6/6 KILLED, none void (mutants2.log): G1 switchTo calls
  onSwitch itself (3 fail), G2 askGenesis calls onCluster itself (1),
  G3 report rethrows (4; mutants2-g3.log, as first written it hit the
  guarded trace and was G4 again), G4 report leaves no trace (2), G5
  the trace is not guarded (1), G6 an empty message is printed empty
  (1). Y1 (A's review: the trace cut at 159, 161 or 1000 characters)
  survived the first version of the tests; KILLED now, 1 fail each
  (r6/y1.log).
- node --test cloud/test/*.test.mjs: 244/244. claims OK, client 244.
  (The red and mutant logs were taken on the earlier base, 5a7c6b2,
  with 243 tests; A's ninth freshness test came in since.)

Sibling search: FailoverRpc has no other callbacks; Rpc has none.
alerter reports its own failures through onError, outside any request.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #129 (859ed7e) went red in CI's `mutants` job: finding 115 moved
murmur3_32 in crates/bulkhead-core/src/floor.rs by 28 lines, and the
murmur3 tail's exclusion is pinned to lines (finding 105), so
`floor\.rs:(327|330|333):11:` matched 0 mutants against a count of 3.
The guard refused as designed, but only in CI: scripts/check.sh ran
tools/mutant_exclusions.py's self-test, not the check on the real
config, so the author's and the reviewer's check.sh were both green.

scripts/mutants.sh gets `--exclusions-only`: the check it already runs
first (the same lines, not a copy), then exit. It takes no other
argument (exit 2), so a --config can't turn it into a silent pass. The
counts are the mutants of the version CI installs, so that version is
read from ci.yml's `cargo install --locked cargo-mutants@X` (exit 2
unless that names one X.Y.Z), and with another version installed, or none, the
mode prints that it did not check and passes. check.sh runs it right
after the claims step (0.3 s). CI's `check` job has no cargo-mutants,
so it passes unchecked there; the `mutants` job still runs the check
first, as before. Both agents' machines have 27.1.0. check.sh's header
and docs/09 (command list, CI table, §6 "In CI.") say so.

Evidence, tasks/exclusions-in-check/r5/ in the autopilot directory:
- check-red.log: check.sh on this tree, rc 1 at the new step, CI's
  message ("matches 0 mutants, its count says 3").
- check-green-with-pattern-fix.log: pattern repointed to (355|358|361)
  in the worktree only: "22 of 800 mutants excluded, each pattern
  matching its count", ALL CHECKS PASSED, rc 0.
- paths.log, `scripts/mutants.sh --exclusions-only` with JOBS=auto:
  stale rc 1; an extra argument rc 2; repointed rc 0; ci.yml edited to
  27.1.1 prints "not checked, they need cargo-mutants 27.1.1" rc 0;
  the install line twice rc 0; 27.1.0-rc.1 rc 2; no version rc 2. Each edit was reverted (git status).

Code review (5 passes) shaped this. Pass 2: the first draft called
`scripts/mutants.sh --list`, which lists again after the check with the
run's options, so JOBS=auto failed check.sh after a passing check. Pass
3: the flag was tested twice and reached cargo-mutants when combined
with others. Pass 4: the version was a fourth copy of CI's; a version
line with a suffix was read as another version. Pass 5: a
pre-release pin was read as its release, and the same install line
twice failed; docs/09 named 27.1.0 again. Kept as is: the step
runs early (fast feedback, like claims); another version passes with a
printed line, since its counts can differ; a full mutants.sh run is not
gated (as before). Not done here: ci.yml's cache key repeats the version (bumping only the
install line would keep CI on the cached old one), cargo-mutants in CI's
`check` job (a
draft PR or the weekly run gets no exclusion check), and the root fix,
a murmur3 tail pattern not pinned to lines (e.g. the tail in its own
function), which edits core code the programs build from (floor.rs:454
calls murmur3_32); proposed to A.

tools/mutant_exclusions.py's self-test runs a copy of mutants.sh and
itself in a temporary tree with a planted config and ci.yml (the repo's
target/ is not written: shasum same before and after), and a planted
`cargo` in $HOME/.cargo/bin that logs its calls, reports a given
version or fails as if missing, and serves two lists. Ten runs:
`--config x` and `--config=x` rc 2, no cargo call; another version, and none, "not
checked" rc 0 after only `--version`; two install lines, none, and a
pre-release, rc 2; the same line twice is one version; CI's
version with lists the config matches rc 0 after exactly the version
call and the two lists (so it stops after the check); with lists it
does not match rc 1. python3 is linked beside the shim, since a pyenv
or asdf python3 can fail under a planted $HOME (not tried here). check.sh now runs
this self-test before the exclusions step, so a broken check fails
under its own name first; docs/09's command list and `check` row follow
the new order.

Mutants of mutants.sh, each applied, red, restored
(tasks/exclusions-in-check/r6/mutants.log; rc 1 each, failing cases in
brackets): M1 `!=` -> `==` on the version test, review F1's first
mutant [4]; M2 the `$#` guard dropped, F1's second [1]; M3 `exit 1`
after the check [1]; M3b that line deleted, so a passing check goes on
to the full run [1]; M4 the skip exits 1 [2]; M5 the version read
always matches [2]; M6 the check's status ignored [1]; M7 only the first
of two install lines read [1]; M8 the X.Y.Z guard deleted [2];
M9 `sort -u` dropped [1]; M10 the X.Y.Z regex loosened to any word
[1]; M11 the guard loosened to `$# < 2` [1]. Restored: self-test
OK, 32 cases.

r2, after A's review (F1: M1 and M2 survived, no in-tree test) and
four more code-review passes. Taken: the first test draft ran the real
mutants.sh and wrote fake lists into the repo's target/ (now a copied
tree); a planted $HOME could break a pyenv python3 (linked); the stop
after a passing check was untested (M3, M3b); the self-test ran after
the check it guards; no-install-line case (M8); the duplicate line and
pre-release cases were tested only out of tree (M9, M10); one extra
argument was untested (M11); docs order,
and docs/09 §3 lists `--exclusions-only`. Not taken:
an install line in a ci.yml comment fails closed, rc 2 naming the
cause; `--exclusions-only` is read only as the first argument (as
documented; elsewhere cargo-mutants refuses the flag); mutants.sh:3
repeats 27.1.0 (it was there before); no TimeoutExpired handling (as
the existing command-line run); the two lists run one after the other;
a tree without ci.yml fails at the step (a git checkout or archive has
it); check.sh rewrites target/mutants-*.txt, as any mutants.sh run in
that tree does (one run per tree at a time).

Sibling search: the only line-pinned pattern in .cargo/mutants.toml is
the murmur3 tail (its header says so). Every other pattern has `\d+`
for the line; limiter.rs's `:(27|26)` is two columns, not lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guardian builds its alerter before checkConfig, so a start refused
for its config still alerts (finding 71). It took alert.format as
given: a config refused for "format": "slak" or "Slack", or for another
misspelt key beside a correct url, posted the JSON body, and a Slack or
Discord webhook does not show it, so the alert saying the guardian did
not start was lost. A config with no format and a Slack URL posted
JSON on every alert the same way.

formatFor (alert.mjs): a valid format is kept; otherwise a Slack
incoming webhook (hooks.slack.com/services/...) gets Slack's body and a
Discord webhook (discord.com or discordapp.com and their subdomains,
/api[/vN]/webhooks/<id>/<token>) Discord's; on any other URL a format
refused only for its case or spacing ("Slack") is read as meant (a
string only: an object format must not throw before the fatal handler
exists); else JSON. Slack workflow triggers (hooks.slack.com/triggers/...) take the
JSON body's keys as variables, so they keep it, and Discord's
Slack-compatible .../slack endpoint is not guessed. docs/07 and
alert.mjs's header say so. checkConfig still refuses "Slack": only the
alert reads it.

Evidence (tasks/alert-format-host/):
- red-old.log: the new end-to-end test (main.mjs with "format": "slak",
  fetch redirected from hooks.slack.com to a local server by a
  --require preload) against the old alert.mjs: the body's keys were
  severity, title, detail, at, protocol, not text.
- mutants.log: 18 mutants of formatFor and its use, all killed by the
  unit test; the end-to-end test also kills the two on its path.
- Review (A): the Slack host's exact compare was unpinned
  (host.endsWith("slack.com") survived); a notslack.com case kills it
  (r6/n1-red.log: fail 1 of 11; r6/green.log 11/11). The Discord
  subdomains, /api/vN and trailing slash the code takes are now in the
  JSDoc and docs/07, and docs/07 says a Discord webhook's /slack or
  /github endpoint gets JSON.
- node --test cloud/test/*.test.mjs 246/246; tools/claims.py OK
  (client 246; docs/09 line 40 updated from 244).

Sibling search: alerter is the only webhook sender with a format
(liveness GETs its URL with no body; cloud/e2e posts to a local hook,
still JSON); main.mjs is its only caller. Not covered: a refused config
whose url key is itself misspelt has no URL to alert (backlog).

Finding 122 (reserved); its docs/09 row waits for the rows batch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finding 123. An alert's detail can hold an RPC's own words: solana.mjs
puts the endpoint's JSON error in the Error's message, and a failed scan
or send logs it as an error line, which alerts. webhookBody passed that
text through as chat markup. In Slack, "<!channel>" from the endpoint
pinged the operators' channel and "<url|label>" showed a link under a
label of the endpoint's choosing, in the channel the operators act from.
In Discord, "@everyone" pinged, "[label](url)" was a masked link, "||"
could hide the rest of the alert (the tx signature) in a spoiler, and a
URL unfurled into a preview. Discord's cut at 2000 UTF-16 units could
also keep half a surrogate pair.

Fix: both chat bodies put the detail in a code block (none when it is
empty), its backticks swapped for ˋ (U+02CB) so it cannot close the
block and half pairs cut upstream (main.mjs cuts errors to 160 or 200
units) for U+FFFD. Slack's text also escapes &, < and > (Slack's control
characters, which code blocks do not disable) and sets unfurl_links and
unfurl_media false. Discord's body sets allowed_mentions { parse: [] }
and flags 4 (SUPPRESS_EMBEDS), and cuts the detail, not the whole text,
so the block stays closed, dropping a high surrogate left at the cut.
The title line is fixed wording and unchanged; the JSON body is
unchanged (its consumer decides). docs/07 says what the chat bodies now
do, and that a bare URL may still show as a link. The two tests that
pin Slack's body keys (alert.test.mjs:123, restart.test.mjs:396) list
the new ones. docs/09's client count 246 -> 247.

Evidence (tasks/alert-text-escape/):
- red-parent.log: the tests against the parent's alert.mjs, fail 2 (the
  new test, and the key list at alert.test.mjs:123).
- mutants.log: 22 mutants (each escape dropped, escape set without &,
  not global, allowed_mentions dropped or parse ["users"], flags dropped
  or 0, unfurl_links dropped, unfurl_media true, no backtick swap, no
  toWellFormed, a block for an empty detail, no surrogate trim, trim
  unanchored or of the low half, room without CLOSE, one over or from
  the title alone (a protocol label pins it), each block dropped) each
  fail the test; restored, pass 12 fail 0. r2/r14.log: trimming either
  half at the cut (A's R14) fails 1/32 on a pair that ends exactly at
  the cut ("d\ud83d" left); restored 32/32, file cmp-equal. An earlier
  draft's `length > room` guard was equivalent under `>=` (the slice is
  then a no-op), so the cut is unconditional.
- check.log: scripts/check.sh ALL CHECKS PASSED.
Not verified against the services (no network): Slack's escaping,
code-block rendering and unfurl keys and Discord's allowed_mentions and
flags are from their documented APIs.

Sibling search: the only other chat or HTML sink is the status page,
whose interpolations of chain or RPC text all go through esc
(index.html:89, :97, :265; the rest are numbers or fixed strings); the
liveness ping is a GET with no body; e2e's fetch is test-only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guardian and the stop cut an RPC's error text by UTF-16 units
(`.slice(0, 160)`, `.slice(0, 200)`). When the cut falls inside a
surrogate pair, the line keeps the first half alone, and JSON.stringify
writes it as "\ud83d". Strict parsers refuse the whole text: serde_json
1.0.151 answers "unexpected end of hex escape" for {"detail":"ab\ud83d"}
and parses {"detail":"ab�"}. So a log shipper or a JSON webhook
receiver written that way drops that log line or alert, and the alert
is retried and then lost (finding 124).

Fix at the writer, not at each cut: jsonText (alert.mjs) makes every
string well-formed (U+FFFD for a half pair) and writes a bigint as its
digits, as main.mjs's replacer did. It writes the guardian's log lines
(main.mjs log, the undelivered-alert line, the state-save line), the
stop's log lines, and every webhook body. The Slack and Discord bodies
were already well-formed (webhookBody, finding 123); the JSON body was
not.

Tests (each fails at the parent):
- alert.test.mjs: jsonText on a half pair, a lone low half, a bigint
  and a whole emoji; the body posted in each ALERT_FORMATS has no
  \ud8xx-\udfxx escape and shows U+FFFD.
- stop.test.mjs: a getGenesisHash error whose emoji's first half is
  unit 159 of the 160 kept; the stop's line shows U+FFFD.
- restart.test.mjs: the same for the guardian's start-up line about a
  backup that does not answer (fakeRpc can now answer `{ error }`).
At the parent: alert 0/1 pass, stop and restart 0/2.
Mutants (tasks/json-well-formed/mutants.py): jsonText without
toWellFormed, without the bigint branch, the body via JSON.stringify,
main.mjs's log via JSON.stringify, stop.mjs's log via JSON.stringify:
5/5 KILLED, restored green. The undelivered-alert and state-save lines
in main.mjs have no test of their own; they use the same jsonText.

Sibling search: every JSON.stringify under cloud/ outside tests. The
others build error text (re-escaped as text, so well-formed), RPC
request bodies from our own values, key and state files, or
deploy-tool output; e2e/attack.mjs's demo log cuts e.message the same
way and is left as is (a demo, not run by operators).

docs/09: client tests 247 -> 250.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant