Skip to content

multi: wait for lnd with a configurable timeout during the SQL migration#1359

Open
ellemouton wants to merge 2 commits into
lightninglabs:masterfrom
ellemouton:kvdb-sql-mig-lnd-ready-timeout
Open

multi: wait for lnd with a configurable timeout during the SQL migration#1359
ellemouton wants to merge 2 commits into
lightninglabs:masterfrom
ellemouton:kvdb-sql-mig-lnd-ready-timeout

Conversation

@ellemouton

Copy link
Copy Markdown
Member

Problem

A node upgraded to litd 0.17.0 (lnd 0.21.1) started up but every API call
returned lnd is not ready for: /lnrpc.Lightning/GetInfo. The litd status
endpoint showed the lit sub-server errored with:

could not create stores: error applying migrations to SQLite store: ...
error listing macaroon IDs when migrating stores to SQL after 120 attempts:
rpc error: code = Unknown desc = the RPC server is in the process of
starting up, but not yet ready to accept calls

Root cause

The kvdb-to-SQL data migration (Mig6ProgrammaticMigration) calls lnd's
ListMacaroonIDs RPC to fetch the macaroon root-key IDs needed for the
firewalldb migration. That RPC is served by lnd's main Lightning server,
which the interceptor rejects with ErrRPCStarting until lnd reaches its
rpcActive state (rpcperms/interceptor.go).

The migration polled this RPC a fixed 120 times with a 500ms delay = 60s.
On a large routing node, lnd took ~76s from wallet-unlock to rpcActive
(16s just to open the main bbolt DB, then building all server subsystems for
a node with ~97k graph channels), so the poll timed out ~16s too early.
The migration failure aborts litd startup; litd stays up but never marks the
LND sub-server ready, so the rpc proxy returns "lnd is not ready for ..." for
every call — even though lnd itself finished starting moments later.

Timeline from the reporter's log:

Time Event
18:01:05 wallet unlocked → migration starts, poll attempt 1/120
18:02:05 migration fails after 120 attempts (60s)
18:02:20 lnd reaches rpcActive (~16s too late)

Because the migration is transactional with ResetVersionOnError, restarting
litd retries it, which is why a restart (warm DB cache → faster lnd) usually
recovers — making this timing/size dependent.

Fix

Replace the fixed attempt cap with a wait bounded by a new, configurable
--lndreadytimeout option, defaulting to a generous 10 minutes (~8x the
observed ~76s). The loop still aborts early on daemon shutdown (ctx
cancellation), and the returned error preserves lnd's real "RPC server is in
the process of starting up" message.

Notes

  • litd's other lnd-readiness waits (setupBasicLNDClient /
    setupFullLNDClient) actually loop indefinitely, bailing only on
    shutdown / lnd-death. I chose a bounded (but generous, configurable)
    timeout here deliberately, because this wait happens inside the migration's
    open SQL write transaction — an unbounded wait there is riskier. Happy to
    switch to unbounded if reviewers prefer matching the client-setup pattern.
  • The ListMacaroonIDs fetch still happens inside the write transaction (as
    before). Moving it out would be a cleaner but larger refactor; left as a
    possible follow-up.

@ellemouton
ellemouton force-pushed the kvdb-sql-mig-lnd-ready-timeout branch from 369afbd to 93f300f Compare July 21, 2026 21:11
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@ellemouton
ellemouton force-pushed the kvdb-sql-mig-lnd-ready-timeout branch from 93f300f to 93188d0 Compare July 21, 2026 21:27
The kvdb-to-SQL data migration polls lnd's ListMacaroonIDs RPC, which
only becomes available once lnd reaches its "RPC active" state. On
nodes with a large channel/graph state, lnd can take well over a
minute to get there after the wallet is unlocked, which exceeded the
previous fixed 60-second (120 x 500ms) poll budget and caused the
migration - and therefore litd startup - to fail permanently,
requiring a manual restart.

Replace the fixed attempt cap with a wait bounded by the new
--lndreadytimeout config option, defaulting to a generous 10 minutes,
while still aborting early if the daemon is shutting down. The wait
happens inside the migration's SQL write transaction, so it is kept
bounded rather than unbounded as a safety backstop.
Document the new --lndreadytimeout option and the startup failure it
fixes in the 0.17.1 release notes.
@ellemouton
ellemouton force-pushed the kvdb-sql-mig-lnd-ready-timeout branch from 93188d0 to 92d13b4 Compare July 21, 2026 21:38
@bitromortac
bitromortac self-requested a review July 22, 2026 10:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants