Skip to content

Comments

fix: deflake //rs/tests/ckbtc:ckbtc_minter_checker#8990

Open
basvandijk wants to merge 4 commits intomasterfrom
ai/deflake-ckbtc_minter_checker
Open

fix: deflake //rs/tests/ckbtc:ckbtc_minter_checker#8990
basvandijk wants to merge 4 commits intomasterfrom
ai/deflake-ckbtc_minter_checker

Conversation

@basvandijk
Copy link
Collaborator

@basvandijk basvandijk commented Feb 22, 2026

//rs/tests/ckbtc:ckbtc_minter_checker has been slightly flaky in the last week:

$ bazel run //ci/githubstats:query -- top 1 flaky% --week --include //rs/tests/ckbtc:ckbtc_minter_checker
...
┍━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━━┯━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━┯━━━━━━━━━━━━┯━━━━━━━━━┯━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━┑
│    │ label                                 │   total │   non_success │   flaky │   timeout │   fail │   non_success% │   flaky% │   timeout% │   fail% │   impact │   total duration │   duration_p90 │ owners   │
┝━━━━┿━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┿━━━━━━━━━┿━━━━━━━━━━━━━━━┿━━━━━━━━━┿━━━━━━━━━━━┿━━━━━━━━┿━━━━━━━━━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━━┿━━━━━━━━━┿━━━━━━━━━━┿━━━━━━━━━━━━━━━━━━┿━━━━━━━━━━━━━━━━┿━━━━━━━━━━┥
│  0 │ //rs/tests/ckbtc:ckbtc_minter_checker │     123 │             3 │       3 │         0 │      0 │            2.4 │      2.4 │          0 │       0 │    25:03 │         17:07:03 │           8:21 │ defi     │
┕━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━┙

Failing due to, what on first sight seems to be, different reasons:

$ bazel run //ci/githubstats:query -- last --flaky --week  //rs/tests/ckbtc:ckbtc_minter_checker --columns=last_started_at,errors
...
╒════╤═════════════════════════╤══════════════════════════════════════════════════════════════════════════════════════════════════════════╕
│    │   last started at (UTC) │ errors per attempt                                                                                       │
╞════╪═════════════════════════╪══════════════════════════════════════════════════════════════════════════════════════════════════════════╡
│  0 │ Fri 2026-02-20 11:35:27 │ 1: -- Test timed out at 2026-02-20 11:35:26 UTC --                                                       │
├────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│  1 │ Wed 2026-02-18 17:27:59 │ 1: test_btc_checker: bug: could not send btc to btc client : JsonRpc(Rpc(RpcError { code: -26, messag... │
├────┼─────────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│  2 │ Wed 2026-02-18 09:30:05 │ 1: test_btc_checker: called `Result::unwrap()` on an `Err` value: JsonRpc(Rpc(RpcError { code: -32603... │
╘════╧═════════════════════════╧══════════════════════════════════════════════════════════════════════════════════════════════════════════╛

Claude Opus 4.6 made the following root cause analysis and accompanying fix:

Root Cause

The RpcClient::send() method used ALL UTXOs from the sender address as transaction inputs, creating oversized transactions. After mining 101+ blocks to the wallet address, dozens of mature coinbase UTXOs were included, resulting in transactions whose minimum relay fee exceeded the hardcoded BITCOIN_NETWORK_TRANSFER_FEE of 1410 satoshis.

This caused three types of flaky failures:

  1. min relay fee not met, 1410 < 5628 (RPC error -26): The fee was too low for the large transaction size.
  2. ProcessNewBlock, block not accepted (RPC error -32603): Downstream effect of transaction pool issues.
  3. Test timeout: Time wasted on failed operations and retries.

Fix

  • Modified RpcClient::send() to select only the minimum UTXOs needed to cover amount + fee, keeping transactions small and fees adequate.
  • Added retry logic (5 attempts with 2s delay) to send_to_btc_address() and generate_blocks() in test utils for transient RPC errors & made the functions idempotent.

See .claude/skills/fix-flaky-tests/SKILL.md.

Root cause: The RpcClient::send() method used ALL UTXOs from the sender
address as transaction inputs, creating oversized transactions. After
mining 101+ blocks to the wallet address, dozens of mature coinbase
UTXOs were included, resulting in transactions whose minimum relay fee
exceeded the hardcoded BITCOIN_NETWORK_TRANSFER_FEE of 1410 satoshis.

This caused three types of flaky failures:
1. 'min relay fee not met, 1410 < 5628' (RPC error -26): The fee was
   too low for the large transaction size.
2. 'ProcessNewBlock, block not accepted' (RPC error -32603): Downstream
   effect of transaction pool issues.
3. Test timeout: Time wasted on failed operations.

Fix:
- Modified RpcClient::send() to select only the minimum UTXOs needed to
  cover amount + fee, keeping transactions small and fees adequate.
- Added retry logic (5 attempts with 2s delay) to send_to_btc_address()
  and generate_blocks() in test utils for transient RPC errors.

See .claude/skills/fix-flaky-tests/SKILL.md
@github-actions github-actions bot added the fix label Feb 22, 2026
@basvandijk basvandijk marked this pull request as ready for review February 22, 2026 13:01
@basvandijk basvandijk requested a review from a team as a code owner February 22, 2026 13:01
@github-actions github-actions bot added the @defi label Feb 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant