Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Bulkhead is a small Solana program that holds a protocol's token accounts on the

**[The attack](#what-it-does-to-an-attack)** · **[How it works](#how-it-works)** · **[Verify it](#verify-it-yourself)** · **[Devnet](#live-on-devnet)** · **[FAQ](docs/FAQ.md)** · **[Docs](#documentation)**

[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-19%2F19-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-154%2F154-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE)
[![CI](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml/badge.svg)](https://github.com/shuv-amp/bulkhead/actions/workflows/ci.yml) [![devnet](https://img.shields.io/badge/devnet-live-57606A?style=flat-square)](#live-on-devnet) [![Kani](https://img.shields.io/badge/Kani_proofs-19%2F19-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![tests](https://img.shields.io/badge/program_tests-163%2F163-57606A?style=flat-square)](docs/09-VERIFICATION.md) [![audit](https://img.shields.io/badge/audit-not_yet,_not_for_real_funds-E5481B?style=flat-square)](SECURITY.md) [![license](https://img.shields.io/badge/license-Apache--2.0-57606A?style=flat-square)](LICENSE)

<br>

Expand Down Expand Up @@ -165,7 +165,7 @@ These scripts install nothing. The core has zero dependencies, and Cloud and the

| Check | Result (latest run of each) |
|---|---|
| **Programs on the real runtime** <br><sub>LiteSVM, mainnet feature set, mainnet token program</sub> | **154/154**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction (7,075 swaps refused, 16 Oct) |
| **Programs on the real runtime** <br><sub>LiteSVM, mainnet feature set, mainnet token program</sub> | **163/163**: every scenario (S1 Drift, S4 bank run, S7, S8, S9, S10 Floor), the negative tests, measured compute, and an account-substitution sweep of every instruction (7,075 swaps refused, 16 Oct) |
| **Bounded proofs** <br><sub>Kani, and exhaustive checks</sub> | **19/19 harnesses verified** on 16 Oct (16 for every value of their types), 240 s of verification ([`09 §5`](docs/09-VERIFICATION.md#5-bounded-proofs-kani-and-exhaustive-checks)). 6 exhaustive tests over ~584M inputs |
| **Differential** <br><sub>an independent Python reference vs the Rust core</sub> | **6,943,561 lines identical** (6 seeds × 200k cases, 27 Sep); 33 commands now, 300k cases on every pull request and 3M weekly in CI |
| **Program vs model** | Random steps on-chain, every outcome equal to the core's prediction: 12,000 (Vault), 300,000 with random limits and every brake (Vault), 200,000 (Floor) |
Expand Down
5 changes: 3 additions & 2 deletions docs/09-VERIFICATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The programs are checked against the core as a model (§7). Two implementations
| **Bounded proofs (Kani)** | 19 harnesses: logic at full width, 3 at small width (§5) | **19/19 verified** (16 Oct) |
| **Coverage-guided fuzzing** | 9 libFuzzer targets (`fuzz/`, cargo-fuzz, overflow checks and AddressSanitizer on): the Token-2022 mint policy, the token-account reader and the guardian-set parser Floor and the Vault share against independent statements of them; the Pyth `PriceUpdateV2` parser against the documented layout; load-store-load round trips of every Vault and Floor account; the loader-v3 parsers and snapshot identity; the limiter at full width (exact `available_now`, instant and queue rules, errors change nothing, window changes keep the used fraction); the ledger (backed ≤ requested, the same amount on a re-ask, `debit_backed` accepts it). Line coverage is measured (`cargo fuzz coverage`); seeds in `fuzz/seeds/` reach what a mutator rarely builds | 6 Oct: 10 min per target, 36M–277M executions each; one harness-oracle crash (finding 40), no program bug; every targeted parser at 100% of lines after the seeds (finding 41). That run had 8 targets: the guardian-set parser's, added 14 Oct, has no recorded 10-minute run or coverage measurement. CI: 60 s per target |
| **Mutation testing** | `cargo-mutants` on the core: 797 mutants (run 10, §6), and in CI on every pull request's changed lines | Every non-equivalent mutant killed; 22 equivalents, each explained and excluded by exact pattern |
| **Program tests** | Floor (50) and Vault (104) on the real agave runtime: LiteSVM, mainnet feature set, the mainnet token program | 154/154 |
| **Program tests** | Floor (50) and Vault (113) on the real agave runtime: LiteSVM, mainnet feature set, the mainnet token program | 163/163 |
| **Account substitution** (`programs/tests/tests/common/sweep.rs`) | For every Vault and Floor instruction (all 30 of the Vault's, tags 0–29, including the read-only views `attest` and `quote_release` and both scopes of `unfreeze`; registration both by the caller PDA and by the upgrade authority; the Floor lifecycle end to end): a transaction that succeeds, then each account swapped in turn for a fresh address, every other account of the transaction, a second market's or a second protected program's accounts, a stranger's token accounts and buffers, other tickets, programs and sysvars, and for a signer every other known key; then each key that signs at any slot (a free one included) and stands at a checked slot: unsigned at every checked slot (a free slot where it signed gets the payer, once a control shows the payer is accepted there; not for the fee payer, which signs anyway), since signing is per key and unsigning one slot of a key that signs at another proves nothing. A key at several slots is otherwise swapped one slot at a time, so a check that only compares those slots with each other is not tested by a consistent attacker key at all of them. Every swap must fail, except at the positions each program leaves free on purpose, each listed with its reason (a rent payer, a ticket account only used when part of a release queues, another guardian of the same set). The first version missed two of four bugs planted to test it (a settle that pays anyone, a cancel of another vault's ticket): it had no look-alike token account or ticket to offer. With them, all four are caught | 16 Oct, with four guardians in the test sets (finding 107; an eject's signers may now be any two of the three others, so each of those slots is free for the third, never for the target or the other signer again; with a guardian listed twice counted as two, the sweep fails): 5,365 Vault and 1,710 Floor substitutions (37 and 18 transactions), refused. 16 Oct, with `attest`, `quote_release` (read-only accounts, quoting an owner's own deposit; as in every release here, the price slot is left free) and the vault-scope `unfreeze`: 5,308 Vault and 1,683 Floor substitutions (37 and 18 transactions), refused. Each of three one-site mutants fails the sweep and passes without those three runs: `attest` not checking its token account, a quote skipping the vault-account check a release makes, and the vault-scope `unfreeze` loading a vault without checking its protocol. 16 Oct, with signing keys tried per key (before, a key that signs at two slots was never tried unsigned: the user of Harbor's deposit and withdraw; with the withdraw's signer check deleted, the sweep now fails): 4,887 Vault and 1,683 Floor substitutions (34 and 18 transactions; one of the 34, "withdraw (signed at the free slot only)", repeats Harbor's withdrawal with the user signing only at a free slot, which adds no program coverage; the test checks that the user was tried unsigned there, so it fails if the sweep stops picking such a key), refused. 10 Oct, with a release to none signed by the owner (finding 95; with that check deleted, the sweep accepts the owner unsigned and fails): 4,499 Vault and 1,683 Floor substitutions (33 and 18 transactions), refused (9 Oct, with `close_buffer`, the signing release and the exit's account 6: 4,508 and 1,549; 8 Oct: 4,474 and 1,472). Found finding 53. Accepted and explained: a veto of a lifeboat exit where none is pending changes nothing (the next proposal clears it) |
| **Program ↔ model differential** | Random operation sequences on-chain vs a host model built only from the core; invariants after every step (§7) | 12,000 steps across 24 seeds, identical; 24,000 steps across 24 seeds after the A1 fix (29 Sep), identical |
| **Wide program differential** | The same, with a limit configuration drawn per seed (floors and caps far above the balance, a closed lane, a queue delay past the horizon) and every instruction that changes what can leave: protocol and vault freezes, quorum unfreezes, ticket extensions, backlog resets, lifeboat moves both ways, limit changes; `quote_release` against the release, every `attest` field against the core (§7) | 5 Oct: 200,000 steps across 400 configurations, then 100,000 with limit changes after finding 30, identical |
Expand Down Expand Up @@ -84,7 +84,7 @@ scripts/mutants.sh # mutation testing, release profile (~45
scripts/fuzz.sh # libFuzzer, every target 60 s in parallel (SECONDS_PER=600 for longer)
(cd fuzz && cargo clippy --all-targets -- -D warnings) # lint the fuzz crate (its own workspace; CI's fuzz job)
scripts/build-programs.sh # Floor, Vault, Harbor for sBPF v3; checks e_flags == 3
(cd programs/tests && cargo test --release) # 154 program tests + the program-vs-model differentials
(cd programs/tests && cargo test --release) # 163 program tests + the program-vs-model differentials
(cd programs/tests && BULKHEAD_PROGRAM_SEEDS=24 BULKHEAD_PROGRAM_STEPS=500 cargo test --release --test differential)
cloud/e2e/run.sh # validator + guardian + attacker
node --test cloud/test/*.test.mjs # client, policy, JS vs Python vs Rust differential (needs the worker)
Expand Down Expand Up @@ -362,6 +362,7 @@ The overflow guards stay in the code as defense in depth; the tests document why
| **Floor differential** (`programs/tests/tests/floor_differential.rs`) | Random heartbeats, vetoes (no id, the pending id, another id, a malformed one), proposals of every kind (some invalid), executes, cancels, ejections, aborts by guardians and owners, approvals, snapshot restores, re-staging and re-pinning, recovery keys (proposed by the owner or by themselves, displacing the owner's proposals, surviving stops, dead once vetoed or expired, passed by a delay raise, replaced by the owner with a quorum, never a guardian), with time jumps that let guardians go stale. A model built from `guardian_alive`, `quorum`, `can_eject`, `can_execute`, `can_restore` and `upgrade_eta` predicts whether each instruction succeeds; after every step the guardian list, heartbeats, owner, recovery key, threshold, interval, delay, status, arming, proposal id, the pending proposal (and who proposed it) and the live program bytes are compared. | 8 Oct: 200,000 steps (400 seeds × 500), **all identical**; each recovery path taken hundreds of times (the rarest, a delay raise past a recovery proposal, 257). The default run (16 seeds × 1,500, in CI) takes the rarest required path, a quorum's snapshot restore, about 20 times |
| **Scenarios** | S1 (G-1 second-for-second, including the EMA reference), S4 (a 20-depositor bank run during a freeze, 100% instant, no prices), S7, S8 (ten payouts in one transaction), S9 (backed exactly as the core computes), S10 (Floor lifecycle) | Pass |
| **Negative tests** | Wrong caller, missing or foreign asset account, wrong feed, frozen settlement, early settlement, wrong recipient, tickets pending, not sealed, import exceeding balance, seal veto, permanent-delegate mint, delay below 72h, abdicated kind, freeze too long, no quorum, admin unfreeze, eject below threshold, and Floor's full set (`06 §6`) | Pass |
| **Vault refusals deleted one at a time** | 16 Oct, on the AUD-2 build: 38 refusals, across 14 of the 18 Vault codes no test asserted by value, deleted one at a time with the whole program suite run against each. Not deleted: the two places that map a core error to `ChangeExecuted` and `ZeroAmount`, `token.rs:233`, and two of the codes, `Overflow` and `LedgerCorrupt` (checked arithmetic, decoding). The other two, `ExtensionTooLong` and `LaneClosed`, the run missed; the review deleted one check of each: the 7-day total on a ticket's extensions (`brakes.rs:86`) and the mapping of a closed lane (`custody.rs:240`) | 13 failed a test, 25 survived. 18 of the 25 survivors now fail a test that asserts the Vault's own code. Among them: another protocol's guardian extending a ticket, a seal that leaves a vault out, invalid limits and assets in a proposed change, an import into a ninth mint, and a quote to an account no token program owns. The review's two also survived, and both now fail such a test: the 7-day total (docs/05 row 11), which the wide differential models (`differential_wide.rs:406`) but whose default run never extends one ticket that far, and a closed lane's code; the same tests pin an extension that moves the ETA earlier, one over 24h, and a lane closed by either bucket. Of the 25's other 7, six are guards no transaction is known to reach: the vault PDA address (`lib.rs:324`; `create_vault` creates a vault only at its PDA), the seal's checks of a lifeboat's owner, mint and authority (`genesis.rs:519`, `:523`; at the seal no lifeboat holds data: only `lifeboat_move` creates one, `move_in` requires the seal (`lifeboat.rs:53`), and the seal happens once), a pending-ticket count below zero (`brakes.rs:50`, `custody.rs:541`; each ticket adds one when created and is closed once), and `GetAccountDataSize`'s return data from an already-checked token program (`token.rs:235`). The seventh, `custody.rs:399` (`debit_backed` of what `backed_amount` allowed, or less after the consent and freeze caps), is **not shown unreachable**: the core property test debits `backed_amount` itself and one more on random ledgers; less is debited only on a one-mint ledger with no liability (`crates/bulkhead-core/tests/units.rs:375`). `backed_amount_is_monotone_in_request` (`properties.rs:606`) asserts `backed(a) == min(a, backed(MAX))` on random ledgers and requests below 2^52 (`:613`), so any amount up to what `backed_amount` allowed is itself backed, and `debit_backed(b)` refuses only when `backed_amount(b) != b` (`ledger.rs:289`) or a new slot would be needed on a full ledger, which `backed_amount` refuses first (`ledger.rs:237`). That is an argument from a property test, not a proof. The one known exception is a collateral value that overflows u128 (`ledger.rs:262`): there the valid amounts need not form an interval, so the consent or freeze cap (`custody.rs:337`, `:347`) can lower the amount to one that is not valid, and `:399` refuses the release (fail closed). The search's own result is always valid: it keeps only amounts it has checked. Five codes no test asserts by value remain: `NotRegistrar` (its one check fails another test), `TicketNotPending` and `LedgerUnsound` (the survivors above), `Overflow` and `LedgerCorrupt` |
| **Floor on a real program** | `scripts/floor-demo.sh <program-id>`: a mainnet program's bytes on a local validator with mainnet's feature set; register, stop with one guardian, restore with two | SPL Stake Pool (1.08 MB, v0): stopped with Floor's code, restored byte-identical (`06 §7`) |
| **Our own SIMD-0500 check** | `scripts/build-programs.sh` reads each built `.so` with the reference's `elf_info` and fails unless `e_flags == 3` | Floor, Vault, Harbor: v3 |
| **Compute** | Measured through Harbor's CPI (`05 §10`) and for Floor (`06 §7`) | Within targets |
Expand Down
11 changes: 11 additions & 0 deletions programs/tests/tests/common/vault.rs
Original file line number Diff line number Diff line change
Expand Up @@ -25,19 +25,26 @@ pub mod verr {
pub const WRONG_ACCOUNT_TYPE: u32 = 8001;
pub const WRONG_ADDRESS: u32 = 8002;
pub const INVALID_INSTRUCTION: u32 = 8000;
pub const ALREADY_INITIALIZED: u32 = 8003;
pub const NOT_ADMIN: u32 = 8005;
pub const MISSING_SIGNATURE: u32 = 8008;
pub const NOT_GUARDIAN: u32 = 8006;
pub const NO_QUORUM: u32 = 8007;
pub const INVALID_GUARDIANS: u32 = 8023;
pub const WRONG_CALLER: u32 = 8009;
pub const WRONG_VAULT: u32 = 8010;
pub const WRONG_MINT: u32 = 8011;
pub const WRONG_TOKEN_PROGRAM: u32 = 8012;
pub const EXTENSION_NOT_ALLOWED: u32 = 8013;
pub const RECIPIENT_OWNER_MISMATCH: u32 = 8014;
pub const BAD_PRICE_ACCOUNT: u32 = 8015;
pub const NOT_SEALED: u32 = 8016;
pub const ALREADY_SEALED: u32 = 8017;
pub const SEAL_NOT_EXECUTABLE: u32 = 8018;
pub const IMPORT_EXCEEDS_BALANCE: u32 = 8019;
pub const VAULT_COUNT_MISMATCH: u32 = 8020;
pub const INVALID_LIMITS: u32 = 8021;
pub const INVALID_ASSET: u32 = 8022;
pub const DELAY_BELOW_MINIMUM: u32 = 8024;
pub const DELAY_ABOVE_MAXIMUM: u32 = 8048;
pub const INSUFFICIENT_BALANCE: u32 = 8049;
Expand All @@ -57,11 +64,15 @@ pub mod verr {
pub const CHANGE_NOT_EXECUTABLE: u32 = 8033;
pub const CHANGE_VETOED: u32 = 8034;
pub const CHANGE_EXPIRED: u32 = 8035;
pub const CHANGE_EXECUTED: u32 = 8036;
pub const ABDICATED: u32 = 8037;
pub const NO_SLOT: u32 = 8040;
pub const DUPLICATE_ACCOUNT: u32 = 8041;
pub const CANNOT_EJECT: u32 = 8042;
pub const GENESIS_ONLY: u32 = 8043;
pub const DESTINATION_NOT_SETTLED: u32 = 8044;
pub const NOTHING_TO_EXIT: u32 = 8045;
pub const ZERO_AMOUNT: u32 = 8046;
}

pub fn so(name: &str) -> Vec<u8> {
Expand Down
Loading