can't close channel in state "CHANNELD_SHUTTING_DOWN"

[martijnbolt]

Issue and Steps to Reproduce

I cannot close a channel that is in a state "CHANNELD_SHUTTING_DOWN" for ages now. For some reason it seems as if the api does not accept this unilateral close and tries to make it bilateral which the nodes can't negotiate, bringing me to a kind of "deadlock" situation. I guess an api call like "froce close" would be good since publishing my commitment tx should always work

Steps to reproduce:
https://imgur.com/a/rYUlEu6

getinfo output

[cdecker]

How long have you been waiting for the closing state to resolve? In recent versions the channel should eventually fall back to a unilateral close automatically. What version are you running?

[martijnbolt]

v0.5.2-2016-11-21-2069-gb8c6365
aka. Bitcoin Savings & Trust Daily Interest II

it's old I guess... I'll install a new version and let you know.

[martijnbolt]

I now run version: v0.6-392-g898655f

This is the problematic channel now:
`

"peers": [
{
"id": "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432",
"connected": false,
"channels": [
{
"state": "AWAITING_UNILATERAL",
"short_channel_id": "506902:2127:1",
"channel_id": "fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c",
"funding_txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
"msatoshi_to_us": 20000000,
"msatoshi_to_us_min": 20000000,
"msatoshi_to_us_max": 20000000,
"msatoshi_total": 20000000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709551615,
"their_channel_reserve_satoshis": 0,
"our_channel_reserve_satoshis": 200,
"spendable_msatoshi": 19800000,
"htlc_minimum_msat": 0,
"their_to_self_delay": 144,
"our_to_self_delay": 144,
"max_accepted_htlcs": 483,
"status": [
"Loaded from database"
],
"in_payments_offered": 0,
"in_msatoshi_offered": 0,
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"out_payments_offered": 0,
"out_msatoshi_offered": 0,
"out_payments_fulfilled": 0,
"out_msatoshi_fulfilled": 0
}
]
},
{
"id": "034a5fdb2df3ce1bfd2c2aca205ce9cfeef1a5f4af21b0b5e81c453080c30d7683",
"connected": false,
"channels": [
{
"state": "CHANNELD_NORMAL",
-----------------snip---------------

`
I'm using the 'close' command to try and close the channel (I tried this with channel_id and peer_id). This hangs my lightning-cli:.

pi@retropie:~/clightning/lightning/lightningd $ sudo /home/pi/lightning/cli/./lightning-cli close fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c | jq [.]
After this I have to restart my daemon and then the CLI responds again but the channel is still there.

debuglog:
`
2018-08-24T14:43:58.293Z lightning_gossipd(20267): UPDATE WIRE_GOSSIP_GET_TXOUT

2018-08-24T14:43:58.306Z lightning_gossipd(20267): Received channel_update for 518494:2113:1 with old time 1526502819

2018-08-24T14:43:58.321Z lightning_gossipd(20267): Received channel_announcement for channel 518510:1692:0

2018-08-24T14:43:58.333Z lightningd(20257):jcon fd 22: Failing close: Channel is in state AWAITING_UNILATERAL
`
Any tips?

[cdecker]

Can you share the output of sqlite3 $HOME/.lightning/lightningd.sqlite3 "select hex(rawtx) from transactions"? That may contain the close transaction, which we can then broadcast again.

[martijnbolt]

thanks! is this what you're after?

root@retropie:~/.lightning# sqlite3 lightningd.sqlite3 "select hex(rawtx) from transactions"
02000000000101FE5B73607D2D9A2BA50A9A6B18D9BF5AB26F61EBF164E1E2C790E8D910A1C57D010000000040993A8000040047304402203D64688221EAC6C2E22487980015EF95C5D39C1BCE35895EA5FE9627E4D0D5D302204F592F9AE7123EE694BA488B015F72861C4DE1B26945753B9094741F404136C5014730440220303B779DE86407A2BBA60235FFDB1BF3102047D7BD6A05E95233F2C0CE49E9BC02202C22978648D6B7C060112D7B488B205DADAD5FD3949CC4CEC07DC723A541C7DE01475221021941C80DD3899CACF8A2171385FC0129C8B6EE1E34DDAC3FEBB0AF84C3F2372D2102EBA9C89D4C3A6A18B9D8B6129A900B788876EF5D61B5BE0E4291AEDD9608649B52AE8CA4B320
02000000000101022A66C4A878FF332FD7EAD8990064782EA8BEF6ACAEC8EBB5B7175F39B6897A0000000000FFFFFFFF0136850100000000001600149F51114DA9CB090643EAE750D335061A5484E170040047304402201C7499EF747278DA2332BDE7C11FF6B7C86031945BB27A57D4BC8BEAA1C1CC8C0220499B2D53BF399D1AD9F6D7F683200C9EC09DC84E8A59FBB1E6564DC5878B1A0601483045022100C88ED13371918D922C377C248E6F2FF3559D8453450338C11C41C51812D4C0FB02201A05DAA86E43E578B731C4D8F7E3ABABA0E54E3E887C22118422437A07A5B9410147522102B09C2BBE3EC738AF90E788051A2B55DF702A5C679E154CEF6868249558E5887721034FDE3C5968D03C9AC5AD0241B719B096CA10EE7AFC736FA150162AB22F782CBC52AE00000000

[cdecker]

Ok, the first transaction is what I was looking for:

{
  "txid": "229a60c362f111fc1b9996f64104bb845adfbb4a3f5ba826ac3e5e37cd903d0c",
  "hash": "cc3dbb6e32679b0993cd41b572f5e166f010b77ad8469498bda55ccaa09f0230",
  "version": 2,
  "size": 271,
  "vsize": 106,
  "locktime": 548643980,
  "vin": [
    {
      "txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
      "vout": 1,
      "scriptSig": {
        "asm": "",
        "hex": ""
      },
      "txinwitness": [
        "",
        "304402203d64688221eac6c2e22487980015ef95c5d39c1bce35895ea5fe9627e4d0d5d302204f592f9ae7123ee694ba488b015f72861c4de1b26945753b9094741f404136c501",
        "30440220303b779de86407a2bba60235ffdb1bf3102047d7bd6a05e95233f2c0ce49e9bc02202c22978648d6b7c060112d7b488b205dadad5fd3949cc4cec07dc723a541c7de01",
        "5221021941c80dd3899cacf8a2171385fc0129c8b6ee1e34ddac3febb0af84c3f2372d2102eba9c89d4c3a6a18b9d8b6129a900b788876ef5d61b5be0e4291aedd9608649b52ae"
      ],
      "sequence": 2151323968
    }
  ],
  "vout": [
  ]
}

The part that is annoying here is that the closing transaction has no output. This is a known (and fixed) issue where the fee would dominate the entire value and no funds would actually be sent back, however a 0 output transaction is invalid to bitcoin clients, and will therefore never confirm.

The version you have there (b8c6365) is very old, that's why it's still producing these close transactions.

We have two options here:

• Upgrade to a newer version, reset the channel state and try to close again. This either returns a minimal amount to you, or it spends everything in fees creating a single op-return output.
• Use dev-forget-channel to simply forget about the channel and not attempt to recover the $1.34, but at least clear the channel from listpeers

[cdecker]

For reference: the commit that fixed this is 45e145d

[martijnbolt]

For some reason this does not work.

    sudo /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c
    { "code" : -32601, "message" : "Unknown command 'dev-forget-channel'" }

[martijnbolt]

I have installed and use a new version now. How do i reset the channel state or how do i fail the channel?

[martijnbolt]

I found another problem on my node, regarding a different much more recent channel. Now using the latest version of clightning:

pi@retropie:~ $ sudo /home/pi/lightning/cli/./lightning-cli connect 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266 23.237.77.11 9735
{
      "id": "0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266"
}
pi@retropie:~ $ sudo /home/pi/lightning/cli/./lightning-cli fundchannel 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266 277288
{ "code" : -1, "message" : "Peer already AWAITING_UNILATERAL" }

It seems multiple channels have the problem on my pi. How to get rid of these failed channels as dev-forget-channel does not seem to be available in v0.6-392-g898655f? (with or without recovering the funds)

[cdecker]

The dev-forget-channel command is a developer command, which can be enabled by recompiling with developer mode turned on:

./configure --enable-developer --disable-valgrind
make

Then restart the node and you should have the dev-forget-channel available.

[martijnbolt]

Thanks. I have tried this but still no luck unfortunately. After I check the results the channel is still there. As the command suggests using 'close' crashes clighting:

pi@retropie:/etc/systemd/system $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
{ "code" : -1, "message" : "This channel has HTLCs attached and it is not safe to forget it. Please use `close` or `dev-fail` instead." }
pi@retropie:/etc/systemd/system $ /home/pi/clightning/lightning/cli/./lightning-cli dev-forget-channel force='true' 02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432
lightning-cli: Expected key=value in '02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432': Success
pi@retropie:/etc/systemd/system $  /home/pi/clightning/lightning/cli/./lightning-cli listpeers | more
{
    "peers": [
    {
      "id": "02f6725f9c1c40333b67faea92fd211c183050f28df32cac3f9d69685fe9665432",
      "connected": false,
      "channels": [
        {
          "state": "AWAITING_UNILATERAL",
          "short_channel_id": "506902:2127:1",
          "channel_id": "fe5b73607d2d9a2ba50a9a6b18d9bf5ab26f61ebf164e1e2c790e8d910a1c57c",
          "funding_txid": "7dc5a110d9e890c7e2e164f1eb616fb25abfd9186b9a0aa52b9a2d7d60735bfe",
          "msatoshi_to_us": 20000000,
          "msatoshi_to_us_min": 20000000,
          "msatoshi_to_us_max": 20000000,
          "msatoshi_total": 20000000, etc etc etc etc etc