multi: wait for lnd with a configurable timeout during the SQL migration#1359
Open
ellemouton wants to merge 2 commits into
Open
multi: wait for lnd with a configurable timeout during the SQL migration#1359ellemouton wants to merge 2 commits into
ellemouton wants to merge 2 commits into
Conversation
ellemouton
force-pushed
the
kvdb-sql-mig-lnd-ready-timeout
branch
from
July 21, 2026 21:11
369afbd to
93f300f
Compare
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
ellemouton
force-pushed
the
kvdb-sql-mig-lnd-ready-timeout
branch
from
July 21, 2026 21:27
93f300f to
93188d0
Compare
The kvdb-to-SQL data migration polls lnd's ListMacaroonIDs RPC, which only becomes available once lnd reaches its "RPC active" state. On nodes with a large channel/graph state, lnd can take well over a minute to get there after the wallet is unlocked, which exceeded the previous fixed 60-second (120 x 500ms) poll budget and caused the migration - and therefore litd startup - to fail permanently, requiring a manual restart. Replace the fixed attempt cap with a wait bounded by the new --lndreadytimeout config option, defaulting to a generous 10 minutes, while still aborting early if the daemon is shutting down. The wait happens inside the migration's SQL write transaction, so it is kept bounded rather than unbounded as a safety backstop.
Document the new --lndreadytimeout option and the startup failure it fixes in the 0.17.1 release notes.
ellemouton
force-pushed
the
kvdb-sql-mig-lnd-ready-timeout
branch
from
July 21, 2026 21:38
93188d0 to
92d13b4
Compare
bitromortac
self-requested a review
July 22, 2026 10:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A node upgraded to litd 0.17.0 (lnd 0.21.1) started up but every API call
returned
lnd is not ready for: /lnrpc.Lightning/GetInfo. The litd statusendpoint showed the
litsub-server errored with:Root cause
The kvdb-to-SQL data migration (
Mig6ProgrammaticMigration) calls lnd'sListMacaroonIDsRPC to fetch the macaroon root-key IDs needed for thefirewalldb migration. That RPC is served by lnd's main Lightning server,
which the interceptor rejects with
ErrRPCStartinguntil lnd reaches itsrpcActivestate (rpcperms/interceptor.go).The migration polled this RPC a fixed 120 times with a 500ms delay = 60s.
On a large routing node, lnd took ~76s from wallet-unlock to
rpcActive(16s just to open the main bbolt DB, then building all server subsystems for
a node with ~97k graph channels), so the poll timed out ~16s too early.
The migration failure aborts litd startup; litd stays up but never marks the
LND sub-server ready, so the rpc proxy returns "lnd is not ready for ..." for
every call — even though lnd itself finished starting moments later.
Timeline from the reporter's log:
rpcActive(~16s too late)Because the migration is transactional with
ResetVersionOnError, restartinglitd retries it, which is why a restart (warm DB cache → faster lnd) usually
recovers — making this timing/size dependent.
Fix
Replace the fixed attempt cap with a wait bounded by a new, configurable
--lndreadytimeoutoption, defaulting to a generous 10 minutes (~8x theobserved ~76s). The loop still aborts early on daemon shutdown (
ctxcancellation), and the returned error preserves lnd's real "RPC server is in
the process of starting up" message.
Notes
setupBasicLNDClient/setupFullLNDClient) actually loop indefinitely, bailing only onshutdown / lnd-death. I chose a bounded (but generous, configurable)
timeout here deliberately, because this wait happens inside the migration's
open SQL write transaction — an unbounded wait there is riskier. Happy to
switch to unbounded if reviewers prefer matching the client-setup pattern.
ListMacaroonIDsfetch still happens inside the write transaction (asbefore). Moving it out would be a cleaner but larger refactor; left as a
possible follow-up.