Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Bulkhead is a small Solana program that holds a protocol's token accounts on the

**[The attack](#what-it-does-to-an-attack)** · **[How it works](#how-it-works)** · **[Verify it](#verify-it-yourself)** · **[Devnet](#live-on-devnet)** · **[FAQ](docs/FAQ.md)** · **[Docs](#documentation)**

[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-19%2F19-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-176%2F176-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE)
[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-19%2F19-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-179%2F179-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE)

<br>

Expand Down Expand Up @@ -165,8 +165,8 @@ These scripts install nothing. The core has zero dependencies, and Cloud and the

| Check | Result (latest run of each) |
|---|---|
| **Programs on the real runtime** <br><sub>LiteSVM, mainnet feature set, mainnet token program</sub> | **176/176**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction (7,075 swaps refused, 16 Oct) |
| **Bounded proofs** <br><sub>Kani, and exhaustive checks</sub> | **19/19 harnesses verified** on 16 Oct (16 for every value of their types), 240 s of verification ([`09 §5`](docs/09-VERIFICATION.md#5-bounded-proofs-kani-and-exhaustive-checks)). 6 exhaustive tests over ~584M inputs |
| **Programs on the real runtime** <br><sub>LiteSVM, mainnet feature set, mainnet token program</sub> | **179/179**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction (7,075 swaps refused, 16 Oct) |
| **Bounded proofs** <br><sub>Kani, and exhaustive checks</sub> | **19/19 harnesses verified** on 16 Oct (16 for every value of their types), 240 s of verification ([`09 §5`](docs/09-VERIFICATION.md#5-bounded-proofs-kani-and-exhaustive-checks)). 7 exhaustive tests over ~588M inputs |
| **Differential** <br><sub>an independent Python reference vs the Rust core</sub> | **6,943,561 lines identical** (6 seeds × 200k cases, 27 Sep); 33 commands now, 300k cases on every pull request and 3M weekly in CI |
| **Program vs model** | Random steps on-chain, every outcome equal to the core's prediction: 12,000 (Vault), 300,000 with random limits and every brake (Vault), 200,000 (Floor) |
| **Mutation testing** <br><sub><code>cargo-mutants</code>: 797 mutants of the core in the last full run, 15 Oct; in CI on every change to the core</sub> | **Every non-equivalent mutant killed**; 22 equivalent mutants, each with the reason it can't change behavior ([`09 §6`](docs/09-VERIFICATION.md#6-mutation-testing)) |
Expand All @@ -175,7 +175,7 @@ These scripts install nothing. The core has zero dependencies, and Cloud and the
| **Golden vectors** <br><sub>every worked example in the docs, exact boundaries, the abort binaries' bytes, loader layouts</sub> | 119/119 identical, Rust = Python |
| **End to end** | [Above](#what-it-does-to-an-attack) |
| **On devnet** | All three programs run the build of PR #85 (15 Oct; this tree's Floor and Vault add finding 107, and Floor findings 115 and 116, not deployed yet), matched byte for byte against CI's build of that commit, upgrade authority checked. **S1 ran there on 1 Oct**: the admin key ordered 1.2M, 100,000 left, the guardian froze and cancelled the other 11. Floor S10 with real transactions: one guardian stops, two restore byte-identical, 17/17 forbidden actions refused ([below](#live-on-devnet)) |
| **What this process found** <br><sub>in our own code and docs</sub> | **116 findings**: real bugs, two overclaimed guarantees, test gaps, deploy-tooling hazards, and the attacks of adversarial reviews and audits. Each is fixed or stated ([`09 §4`](docs/09-VERIFICATION.md#4-what-this-process-found-every-item-fixed-or-stated)); 96–116 were recorded after the submission (107 is the engagement's AUD-2, decided 16 Oct): 96–99, 101 and 103 are fixed and run on devnet since 15 Oct; 105 (CI tooling), 106, 108, 112 and 113 (Vault tests), 107 (the guardian margin, not yet on devnet), 109 (fuzz targets), 110 and 111 (Floor tests), 114 (the guardian's startup checks, not yet on devnet), 115 and 116 (Floor's snapshot length rule, not yet on devnet; 116's class below sBPF v3 is stated) are fixed; 100, 102 and 104 are stated. The Rust core and the Python reference shared one bug; only a third, independent statement of the rule caught it. |
| **What this process found** <br><sub>in our own code and docs</sub> | **130 findings**: real bugs, two overclaimed guarantees, test gaps, deploy-tooling hazards, and the attacks of adversarial reviews and audits. Each is fixed or stated ([`09 §4`](docs/09-VERIFICATION.md#4-what-this-process-found-every-item-fixed-or-stated)); 96–130 were recorded after the submission (107 is the engagement's AUD-2, decided 16 Oct): 96–99, 101 and 103 are fixed and run on devnet since 15 Oct; 105 (CI tooling), 106, 108, 112, 113 and 120 (Vault tests), 107 (the guardian margin, not yet on devnet), 109 (fuzz targets), 110 and 111 (Floor tests), 114 (the guardian's startup checks, not yet on devnet), 115 and 116 (Floor's snapshot length rule, not yet on devnet; 116's class below sBPF v3 is stated), 117–119, 121–126 and 128–130 (the guardian, its alerts and the emergency stop, not yet on devnet), and 127 (Floor and Vault tests) are fixed; 100, 102 and 104 are stated. The Rust core and the Python reference shared one bug; only a third, independent statement of the rule caught it. |

## Live on devnet

Expand Down
99 changes: 88 additions & 11 deletions cloud/guardian/alert.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -37,36 +37,110 @@ const URGENT_WARNINGS = [
/listed transactions at or before the start slot/,
];

/**
* The most of a line's message (in UTF-16 units) an alert's detail shows: a
* longer one shows its first MESSAGE_MAX - MESSAGE_TAIL - 1 units, "…" and
* its last MESSAGE_TAIL, one fewer on a side whose cut would split a pair.
*/
export const MESSAGE_MAX = 1000;
export const MESSAGE_TAIL = 300;

/** The first `n` UTF-16 units of `s`, less a high surrogate left without its pair. */
function firstUnits(s, n) {
return s.slice(0, n).replace(/[\uD800-\uDBFF]$/, "");
}

/** The last `n` UTF-16 units of `s`, less a low surrogate left without its pair. */
function lastUnits(s, n) {
return s.slice(-n).replace(/^[\uDC00-\uDFFF]/, "");
}

/**
* What made `e` fail, for a log line's `errorCause`, which alertFor keys its
* repeats on in place of the message's first 60 units. Those held no cause:
* every refusal an RPC sends starts `sendTransaction: {"code":-32002,
* "message":"Transaction simul`, so a second cause was muted; and a signed
* transaction's failure starts `transaction <signature>`, new on every
* retry, so a freeze failing on chain alerted on every poll.
* - A transaction's error, however it came (a node's refused preflight, the
* transaction failing on chain, the freeze's own simulation): the error,
* and for an instruction's, only what it failed with. Its index moves with
* the compute-budget instructions sent before it, which a simulation or a
* send without a budget leaves out.
* - Any other RPC error: the method, the code and the text with each run of
* digits as "#", in a word too (a node "behind by 42 slots" is behind by
* 43 a poll later; "retry in 734ms"), but a program's "0x1770" is kept.
* - A signed transaction's failure: its message without the signature.
* Cut at 160 units. Undefined for anything else.
*/
export function errorCause(e) {
return rawCause(e)?.slice(0, 160);
}
function rawCause(e) {
const err = e?.rpcError?.data?.err ?? e?.onChainErr ?? e?.simulatedErr;
if (err != null) {
const ie = err.InstructionError;
return Array.isArray(ie) ? `instruction error ${JSON.stringify(ie[1])}` : `transaction error ${JSON.stringify(err)}`;
}
const r = e?.rpcError;
if (r?.code != null) return `${e.rpcMethod ?? "?"} RPC ${r.code} ${String(r.message ?? "").replace(/(0x[0-9a-f]+)|\d+/gi, (m, hex) => hex ?? "#")}`;
const sig = e?.signature;
const message = String(e?.message ?? "");
if (typeof sig === "string" && sig && message.includes(sig)) return message.replaceAll(sig, "<signature>");
return undefined;
}

/** Actions sent on chain: each one is news. */
const ACTIONS = new Set(["freeze", "cancel", "veto_change", "veto_exit", "floor_veto", "execute"]);

/**
* The alert for one log line, or null. `line` is the object the guardian
* logs. A freeze that landed always alerts. Anything else repeats at most once
* per `windowMs` for the same kind (`seen` remembers when each kind was last
* sent); a kind is what an action acts on (a change, a proposal, a mint), so
* two different vetoes are two alerts, while the cancels of one sweep are one.
* A failed freeze, retried every poll, alerts at most once a `failedFreezeMs`
* per cause. A line that skipped its action is not an alert.
* sent); a kind is what an action acts on (a change, a proposal, a mint,
* the endpoints a failover moved between) and why it failed (the line's
* `errorCause` when it has one, or else its message's first 60 units), so
* two different vetoes are two alerts, while the cancels of one sweep are
* one. A failed freeze, retried every poll, alerts at most once a
* `failedFreezeMs` per cause. A line that skipped its action is not an
* alert.
*/
export function alertFor(line, seen, nowMs, { windowMs = 600_000, failedFreezeMs = 60_000 } = {}) {
const action = line.action;
if (line.skipped) return null;
const level = line.level ?? "info";
const message = line.msg ?? line.error ?? "";
// Its message makes it urgent and keys it; the detail shows both (below).
const texts = [line.msg, line.error].filter((s) => s != null && s !== "");
const message = texts[0] ?? "";
// `notify`: an info line people should still hear about (a Floor restore,
// a restore approval; finding 76). It alerts as a warning.
const urgent = level === "error" || ACTIONS.has(action) || line.notify === true || (level === "warn" && URGENT_WARNINGS.some((r) => r.test(message)));
if (!urgent) return null;
const parts = about(line);
const subject = parts.map((p) => p.shown).join("/");
const kind = `${level}:${action ?? ""}:${subject}:${message.slice(0, 60)}`;
const kind = `${level}:${action ?? ""}:${subject}:${line.errorCause ?? message.slice(0, 60)}`;
const landedFreeze = action === "freeze" && line.signature;
const window = action === "freeze" ? failedFreezeMs : windowMs;
if (!landedFreeze && nowMs - (seen.get(kind) ?? -Infinity) < window) return null;
seen.set(kind, nowMs);
const what = action ? `${action}${line.signature ? " sent" : " failed"}` : level === "info" ? "notice" : level;
const detail = [line.reason, message, ...named(parts, [line.reason ?? "", message]), line.ticketId != null ? `ticket ${line.ticketId}` : null, line.signature ? `tx ${line.signature}` : null]
// An RPC's error text can run to kilobytes (a failed send quotes the
// simulation's logs), and webhookBody cuts Discord's body to Discord's
// 2000-unit limit, so the change, program, mint or ticket after it were
// lost. The detail keeps its start (the RPC's message) and its end (the
// last logs, or what a line says after the error it quotes). A line with
// a message and the RPC's `error` shows both (finding 128): the alert used
// to say what failed but not why.
const said = texts.join(" | ");
let shown = said, heard = [said];
if (said.length > MESSAGE_MAX) {
const [head, tail] = [firstUnits(said, MESSAGE_MAX - MESSAGE_TAIL - 1), lastUnits(said, MESSAGE_TAIL)];
shown = `${head}…${tail}`;
// A word at a cut does not name a part: it reads as cut ("change 4…"
// may be change 45), so that part is named again in full.
heard = [head.replace(/\w+$/, ""), tail.replace(/^\w+/, "")];
}
const detail = [line.reason, shown, ...named(parts, [line.reason ?? "", ...heard]), line.ticketId != null ? `ticket ${line.ticketId}` : null, line.signature ? `tx ${line.signature}` : null]
.filter(Boolean).join(" | ");
const alert = { severity: level === "error" || action === "freeze" ? "critical" : "warning", title: `Bulkhead guardian: ${what}`, detail, at: line.t };
// Not part of the message: lets the sender un-mute an alert it could not deliver.
Expand All @@ -79,15 +153,18 @@ export function alertFor(line, seen, nowMs, { windowMs = 600_000, failedFreezeMs
* word that says it is already in a text (`needle`). alertFor keys its
* repeats on the shown parts ("change 4" and "proposal 4" differ) and names
* them in the detail, so the webhook never keys on a part it doesn't show.
* A program or mint is said by its bare key, and a change or proposal in any
* case ("Change 4"). `what` and `target` may hold a key, so they and the keys
* match exactly: base58 case matters.
* A program, mint or endpoint is said bare, and a change or proposal in any
* case ("Change 4"). An endpoint is an entry's name (`host (rpc)`, `host
* (rpcs[0])`), never its URL, which may hold an API key. `what` and
* `target` may hold a key, so they and the keys match exactly: base58 case
* matters.
*/
function about(line) {
return [
[line.what, ""], [line.target, "on "],
[line.changeId, "change ", false, true], [line.proposalId, "proposal ", false, true],
[line.program, "program ", true], [line.mint, "mint ", true],
[line.endpointFrom, "from ", true], [line.endpoint, "endpoint ", true],
].filter(([v]) => v != null && v !== "") // says() never ends on an empty word
.map(([v, label, bare = false, fold = false]) => ({ shown: `${label}${v}`, needle: bare ? String(v) : `${label}${v}`, fold }));
}
Expand Down Expand Up @@ -247,7 +324,7 @@ export function webhookBody(alert, format = "json", protocol = "") {
const block = (d) => (d ? `${OPEN}${d}${CLOSE}` : "");
if (format === "slack") return { text: `${head}${block(detail)}`.replace(/[&<>]/g, (c) => SLACK_ESCAPES[c]), unfurl_links: false, unfurl_media: false };
if (format === "discord") {
const cut = detail.slice(0, 2000 - head.length - OPEN.length - CLOSE.length).replace(/[\uD800-\uDBFF]$/, "");
const cut = firstUnits(detail, 2000 - head.length - OPEN.length - CLOSE.length);
return { content: `${head}${block(cut)}`, allowed_mentions: { parse: [] }, flags: 4 };
}
return { ...alert, protocol };
Expand Down
Loading