[bug]: pending open channel, opening transaction never confirmed

[niko-mit-k]

Background

I'm running LND 0.18.0-beta on top of Bitcoin Core 27.0.0, running on an Umbrel Home with umbrelOS 1.2.2.
2 channels has been pending on opening for over 2,5 month, it tried to open when fees were very low 1sat/vb

The txid in the channel point can't be found through blockexplorers like mempool.space.

I do already a wallet rescan (reset-wallet-transactions=true), but it does't fix the issue.

I tried using closechannel but that did not work, because the channel is not open. Here are the relevant log entries and output from pendingchannels.

$ lncli pendingchannels
{
    "total_limbo_balance": "0",
    "pending_open_channels": [
        {
            "channel": {
                "remote_node_pub": "xxx",
                "channel_point": "xxx:1",
                "capacity": "600000",
                "local_balance": "599653",
                "remote_balance": "0",
                "local_chan_reserve_sat": "60000",
                "remote_chan_reserve_sat": "60000",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": ""
            },
            "confirmation_height": 0,
            "commit_fee": "2810",
            "commit_weight": "772",
            "fee_per_kw": "2500"
        },
        {
            "channel": {
                "remote_node_pub": "xxx",
                "channel_point": "xxx:1",
                "capacity": "800000",
                "local_balance": "799653",
                "remote_balance": "0",
                "local_chan_reserve_sat": "80000",
                "remote_chan_reserve_sat": "80000",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": ""
            },
            "confirmation_height": 0,
            "commit_fee": "2810",
            "commit_weight": "772",
            "fee_per_kw": "2500"
        }
    ],
    "pending_closing_channels": [
    ],
    "pending_force_closing_channels": [
    ],
    "waiting_close_channels": [
    ]
}

Your environment

• LND "version": "0.18.0-beta commit=v0.18.0-beta",
• umbrelOS 1.2.2 - Linux 81a3ed06c997 6.1.0-23-amd64 Fix name typo in README #1 SMP PREEMPT_DYNAMIC Debian 6.1.99-1 (2024-07-15) x86_64 GNU/Linux
• Bitcoin Core 27.0.0

Steps to reproduce

Open a new channel in Ride The Lightning (RTL) with unconfirmed UTXOs. (Spend Unconfirmed Output)

Expected behaviour

I have successfully opened many channels before, so I am not sure how to reproduce this issue. I have always followed the same process when opening channels, so I am not sure what might be causing this problem.

Actual behaviour

When I run the lncli pendingchannels command, I am able to see information about the channel, but none of the transaction IDs displayed are in the mempool.

Pending open channel, opening transaction never confirmed.

Logs

$ cat ~/umbrel/app-data/lightning/data/lnd/logs/bitcoin/mainnet/lnd.log | grep 'xxx' | tail
2024-08-28 12:13:06.163 [WRN] BTWL: Transaction xxx not accepted by mempool: missing-inputs
2024-08-28 12:13:06.164 [ERR] FNDG: Unable to rebroadcast funding tx xxx for ChannelPoint(xxx:1): transaction rejected: output already spent
2024-08-28 12:13:06.168 [INF] NTFN: New confirmation subscription: conf_id=1, txid=xxx, num_confs=6 height_hint=858794
2024-08-28 12:13:06.168 [INF] FNDG: Waiting for funding tx (xxx) to reach 6 confirmations
2024-08-28 12:13:06.812 [INF] CNCT: Close observer for ChannelPoint(xxx:1) active
2024-08-28 12:13:07.165 [INF] NTFN: Historical spend dispatch finished for request outpoint=xxx:1, script=0 xxx (start=858794 end=858795) with details: <nil>
2024-08-28 12:13:07.948 [INF] CNCT: ChannelArbitrator(xxx:1): starting state=StateDefault, trigger=chainTrigger, triggerHeight=858795
2024-08-28 12:13:09.133 [WRN] CHBU: Replacing disk backup for ChannelPoint(xxx:1) w/ newer version
2024-08-28 12:13:11.396 [INF] PEER: Peer(xxx): Loading ChannelPoint(xxx:1), isPending=true
2024-08-28 12:13:11.409 [WRN] PEER: Peer(xxx): Unable to find our forwarding policy for channel xxx:1, using default values

$ cat ~/umbrel/app-data/lightning/data/lnd/logs/bitcoin/mainnet/lnd.log | grep 'yyy' | tail
2024-08-28 12:13:06.229 [WRN] BTWL: Transaction yyy not accepted by mempool: missing-inputs
2024-08-28 12:13:06.229 [ERR] FNDG: Unable to rebroadcast funding tx yyy for ChannelPoint(yyy:1): transaction rejected: output already spent
2024-08-28 12:13:06.244 [INF] NTFN: New confirmation subscription: conf_id=2, txid=yyy, num_confs=3 height_hint=858794
2024-08-28 12:13:06.248 [INF] FNDG: Waiting for funding tx (yyy) to reach 3 confirmations
2024-08-28 12:13:06.814 [INF] CNCT: Close observer for ChannelPoint(yyy:1) active
2024-08-28 12:13:07.165 [INF] NTFN: Historical spend dispatch finished for request outpoint=yyy:1, script=0 yyy (start=858794 end=858795) with details: <nil>
2024-08-28 12:13:08.024 [INF] CNCT: ChannelArbitrator(yyy:1): starting state=StateDefault, trigger=chainTrigger, triggerHeight=858795
2024-08-28 12:13:09.132 [WRN] CHBU: Replacing disk backup for ChannelPoint(yyy:1) w/ newer version
2024-08-28 12:13:11.471 [INF] PEER: Peer(yyy): Loading ChannelPoint(yyy:1), isPending=true
2024-08-28 12:13:11.471 [WRN] PEER: Peer(yyy): Unable to find our forwarding policy for channel yyy:1, using default values

[ViktorTigerstrom]

Hi @niko-mit-k!

As these specific channels will be unknown for the remote party, even if they are eventually confirmed, you would want to ensure that they do not confirm.

1. In order to do that, can you check which input's the transactions to open the channels were using? You'd want to ensure that you spend at least one of the inputs per channel, in a new transaction, with a higher fee. That will ensure that the channel open transactions will never confirm.

2. When you've spent the inputs, you would want to abandon them in lnd by using the lncli abandonchannel command. Only do this after you've already double spent the inputs for the channel open transactions, as you else might risk that the channel open transaction confirms after the channel has been abandoned, and neither your's or the remote parties' node would know of the channel.

[niko-mit-k]

In order to do that, can you check which input's the transactions to open the channels were using?

How can I do this? In my local mempool.space I don’t see the 2 tx…

[ziggie1984]

if you are not sure whether this outputs are already spent do the following:

Force-Close both channels:

lncli closechannel --force --chan_point XXXX..:Y

=> this should bring the channel into waiting close check with

lncli pendingchannels --include_raw_tx

This should give you the closing transaction of the channel:

Example:

   "waiting_close_channels": [
        {
            "channel": {
                "remote_node_pub": "03d832cbf73db15a167f9a8eca330d79b5a8a35129c80736d29cdf6619a91d793b",
                "channel_point": "33788e65cadb75baddb76e5e6d57b345f1063ecf3ece5d4998936b7e60c182b0:0",
                "capacity": "1000000",
                "local_balance": "498496",
                "remote_balance": "500000",
                "local_chan_reserve_sat": "10000",
                "remote_chan_reserve_sat": "10000",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private": false,
                "memo": ""
            },
            "limbo_balance": "498496",
            "commitments": {
                "local_txid": "463ade40a15b3bec102e714cf2203c89a20d57a1d2ab554e2baa25cb96fc2e45",
                "remote_txid": "371585584e786c9815de3774db9a9937ee902437cd08e47baa0deed2a62d7b27",
                "remote_pending_txid": "",
                "local_commit_fee_sat": "844",
                "remote_commit_fee_sat": "844",
                "remote_pending_commit_fee_sat": "0"
            },
            "closing_txid": "463ade40a15b3bec102e714cf2203c89a20d57a1d2ab554e2baa25cb96fc2e45",
            "closing_tx_hex": "02000000000101b082c1607e6b9398495dce3ecf3e06f145b3576d5e6eb7ddba75dbca658e78330000000000bcab8e80044a0100000000000022002066155294703b42cebeae2ffddea64c731abb9230297ca61383e2b7586a972dc24a01000000000000220020a0b390b4b7d017590f2bd4ac94f4b5ecf4750eec2d1cd155c56344c8d054b80c409b0700000000002200206c576ee8c882157c783f4b3bb7275fa95b8690c7431453857b0e2261b740fd1520a10700000000002200204e7653565a446e62722a7178560eafbf695deab5ae09a9cd53a0ca99a3cb553d040047304402201cf6c6c385d1a022385b12b7cf0c5b7e4df487bbbbf89b478e5703a5a8b4a801022039b2c7e295b84c140dbbbb7e3f20a8a0b357533e9350fa6aa5466c5d36d0de63014830450221008174200ca4eb0087fb3c59ac62497ea1c076daed4f6e96a9fd7b50a053b7ba8302207c48a0daf3ec3424fcb50a15cec9daac47fb675ab26448c7a159292b78b1be9201475221025eafb38641d9b72563badebc401af110ed17e0711f2920f130eb38f19982fb29210390e968ab38647cffe9a360f23ca22260b423bffc26d377dff4c43f5e3d0c9fcc52ae98e98720"
        }

Take this transaction and try to broadcast it in several explores. If this does not work abandon the channel with

lncli abandonchannel

[ziggie1984]

Closing this issue, because this is normal behaviour the counter party will forget about your channel after 2016 blocks.

[ViktorTigerstrom]

Thanks for the extra details @ziggie1984!

Just adding that you should probably see the inputs of the open channel transaction (funding transaction) by running lncli wallet gettx TXID, and the look for the previous_outpoints in that info. The "TXID" is the channel_point in the pendingchannels response, with the :1 part removed.

If you'd want to go the route of double spending the tx(es) inputs, there's a command for that in chantools (https://github.com/lightninglabs/chantools/), which is another project by Lightning Labs. The command is called doublespendinputs, and you can read more about it here: https://github.com/lightninglabs/chantools/blob/master/doc/chantools_doublespendinputs.md.

[niko-mit-k]

Thank you for your help!
Now the two channels are in waiting_close_channels.

--

lncli wallet gettx TXID doesn't work.

No help topic for 'gettx'

--

lncli pendingchannels --include_raw_tx doesn't work.

Incorrect Usage: flag provided but not defined: -include_raw_tx

How can i get the raw tx data for broadcasting?
-> i have only the closing_txid

--

lncli wallet bumpfee doesn't work:
lncli wallet bumpfee --conf_target 1 --force xxx:1

[lncli] rpc error: code = Unknown desc = the passed output does not belong to the wallet

[ViktorTigerstrom]

lncli wallet gettx TXID doesn't work.
lncli pendingchannels --include_raw_tx doesn't work.

Are you sure you're using lnd v0.18.0-beta and the lncli from that version? They should exist.

lncli wallet bumpfee doesn't work:

There's another command for force closure transactions specifically called:
lncli wallet bumpforceclosefee

[niko-mit-k]

I had used the Lightning Shell, but it doesn't work. I think its the wrong version...

Now i try the real shell and it works:

$ sudo docker exec lightning_lnd_1 lncli pendingchannels --include_raw_tx
{
    "total_limbo_balance":  "1399306",
    "pending_open_channels":  [],
    "pending_closing_channels":  [],
    "pending_force_closing_channels":  [],
    "waiting_close_channels":  [
        {
            "channel":  {
                "remote_node_pub":  "xxx",
                "channel_point":  "xxx:1",
                "capacity":  "600000",
                "local_balance":  "599653",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "60000",
                "remote_chan_reserve_sat":  "60000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "599653",
            "commitments":  {
                "local_txid":  "xxx",
                "remote_txid":  "xxx",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "2810",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "xxx",
            "closing_tx_hex":  "xxx"
        },
        {
            "channel":  {
                "remote_node_pub":  "yyy",
                "channel_point":  "yyy:1",
                "capacity":  "800000",
                "local_balance":  "799653",
                "remote_balance":  "0",
                "local_chan_reserve_sat":  "80000",
                "remote_chan_reserve_sat":  "80000",
                "initiator":  "INITIATOR_LOCAL",
                "commitment_type":  "ANCHORS",
                "num_forwarding_packages":  "0",
                "chan_status_flags":  "ChanStatusBorked|ChanStatusCommitBroadcasted|ChanStatusLocalCloseInitiator",
                "private":  false,
                "memo":  ""
            },
            "limbo_balance":  "799653",
            "commitments":  {
                "local_txid":  "yyy",
                "remote_txid":  "yyy",
                "remote_pending_txid":  "",
                "local_commit_fee_sat":  "2810",
                "remote_commit_fee_sat":  "2810",
                "remote_pending_commit_fee_sat":  "0"
            },
            "closing_txid":  "yyy",
            "closing_tx_hex":  "yyy"
        }
    ]
}

I tried to broadcast the closing_tx_hex , but i got an error

Failed to broadcast transaction, reason: bad-txns-inputs-missingorspent

$ sudo docker exec lightning_lnd_1 lncli wallet gettx xxx

[lncli] rpc error: code = Unknown desc = can not find transaction: txid xxx

$ sudo docker exec lightning_lnd_1 lncli wallet gettx yyy

[lncli] rpc error: code = Unknown desc = can not find transaction: txid yyy

Q: I see i have Sats in Limbo limbo_balance. Are the Sats real or fake?

How can i recover these Sats?

Thx for your Help!

[niko-mit-k]

Raw transaction decoding:

• https://www.blockchain.com/en/explorer/assets/btc/decode-transaction

{
  "version": 2,
  "locktime": 539589907,
  "ins": [
    {
      "n": 1,
      "script": {
        "asm": "",
        "hex": ""
      },
      "sequence": 2158552483,
      "txid": "xxx",
      "witness": [
        "",
        "xxx",
        "xxx",
        "xxx"
      ]
    }
  ],
  "outs": [
    {
      "n": 0,
      "script": {
        "addresses": [
          "xxx"
        ],
        "asm": "OP_0 xxx",
        "hex": "xxx"
      },
      "value": 330
    },
    {
      "n": 1,
      "script": {
        "addresses": [
          "xxx"
        ],
        "asm": "OP_0 xxx",
        "hex": "xxx"
      },
      "value": 599653
    }
  ],
  "hash": "xxx",
  "txid": "xxx"
}

Both outs wallet addresses are empty...

[ziggie1984]

These means those channels were never opened, and you can securely abandon it as viktor mentioned (keep the force-close transaction as a backup just to be sure that in case a channel opening get confirmed you can force-close it.

lncli abandonchannel

As you mentioned correctly those funds tagged as pending, are not really there so there is nothing missing in your wallet.