Node crashes probably while trying to make a payment

Issue and Steps to Reproduce

EDIT:

My node often crashes in the same way when I make payments to coinpanic.com and other websites,.

I don't see a crash report in .lightning but the output of lightningd can be found here

getinfo output

{
  "id": "02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0", 
  "alias": "Bight.nl", 
  "color": "23354d", 
  "address": [
    {
      "type": "ipv4", 
      "address": "82.217.214.215", 
      "port": 9735
    }
  ], 
  "binding": [
    {
      "type": "ipv4", 
      "address": "192.168.178.20", 
      "port": 9735
    }
  ], 
  "version": "v0.6-310-g9d8b3a0", 
  "blockheight": 536104, 
  "network": "bitcoin"
}

[cdecker]

I can't retrieve the log, can you post it to a public place?

My server was probably really slow when you tried, could you try again? Anyways, you can find out.txt here as well (or do you mean another file?)

[cdecker]

So the relevant lines seem to be the following:

2018-08-10T13:44:26.760Z lightningd(32312): pay(0x2a89178): sendpay via route: us -> 533632:1635:1 (1502785msat, 614blk) -> 0394396e6e8cf8d9f91de4269dfd2e8416b93b91cae6139057c7283b463e4bfa31 -> 534461:1174:0 (1501770msat, 600blk) -> 03c436af41160a355fc1ed230a64f6a64bcbd2ae50f12171d1318f9782602be601 -> 534588:1980:1 (1501765msat, 456blk) -> 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4fb8 -> 532286:1959:1 (1501760msat, 312blk) -> 024bda1ec91b277d311aa60d04b1fb856efcc7897c876746b680c1f985fe973f73 -> 532608:524:0 (1501760msat, 168blk) -> 02cdf83ef8e45908b1092125d25c68dcec7751ca8d39f557775cd842e5bc127469 -> 536081:990:0 (1500759msat, 24blk) -> 0229cf81c21bbd21c2a41a4ae645933b89bb6d9a5920ca90e41ba270666879adab -> 536081:991:0 (1500759msat, 10blk) -> 02bd3ec3b4336f2f0728743734d5000f4476fcd3d9abc5cc95cacd39c4a4c6e1a0
2018-08-10T13:44:26.801Z lightningd(32312): Sending 1502785 over 7 hops to deliver 1500000
2018-08-10T13:44:26.830Z lightningd(32312): pay(0x2a89178): Try another route
2018-08-10T13:44:27.479Z lightningd(32312): pay(0x2a89178): sendpay via route: us -> 532656:2413:1 (1505105msat, 312blk) -> 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028 -> 532130:1901:1 (1504104msat, 168blk) -> 02cdf83ef8e45908b1092125d25c68dcec7751ca8d39f557775cd842e5bc127469 -> 536081:990:0 (1503103msat, 24blk) -> 0229cf81c21bbd21c2a41a4ae645933b89bb6d9a5920ca90e41ba270666879adab -> 536081:991:0 (1503103msat, 10blk) -> 02bd3ec3b4336f2f0728743734d5000f4476fcd3d9abc5cc95cacd39c4a4c6e1a0
2018-08-10T13:44:27.483Z lightningd(32312): Sending 1505105 over 4 hops to deliver 1500000
2018-08-10T13:44:29.525Z lightningd(32312): 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028 chan #34: htlc 78 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2018-08-10T13:44:29.525Z lightningd(32312): pay(0x2a89178): Try another route
lightningd: Outstanding taken pointers: (nil)
lightning_openingd: lightning_openingd: Writing out status 65522lightning_channeld: Writing out status 65522: Writing out status 65522: : Broken pipe
Broken pipe
lightning_channeld: Broken pipe
Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_openingd: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_openingd: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe

The line lightningd: Outstanding taken pointers: (nil) seems to indicate a false positive in the memory leak detection.

This is the output of the previous crash (with a somewhat earlier version of the software but not much). Maybe that extra information is useful as well:

2018-08-10T12:37:14.238Z lightningd(5892): pay(0x2decc08): Try another route
2018-08-10T12:37:14.974Z lightningd(5892): pay(0x2decc08): sendpay via route: us -> 531808:300:1 (1502616msat, 470blk) -> 0250373555232cec757ea141273e75381c84cc3ab22f1e3a3ca5a2dc55b1904c24 -> 533960:1065:0 (1501601msat, 456blk) -> 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4fb8 -> 532286:1959:1 (1501596msat, 312blk) -> 024bda1ec91b277d311aa60d04b1fb856efcc7897c876746b680c1f985fe973f73 -> 532608:524:0 (1501596msat, 168blk) -> 02cdf83ef8e45908b1092125d25c68dcec7751ca8d39f557775cd842e5bc127469 -> 536081:990:0 (1500595msat, 24blk) -> 0229cf81c21bbd21c2a41a4ae645933b89bb6d9a5920ca90e41ba270666879adab -> 536081:1007:0 (1500595msat, 10blk) -> 0211887ec50c8e7bb4bdb7259281a2314a6bdb70729f52845937f868fa44ad3748
2018-08-10T12:37:14.980Z lightningd(5892): Sending 1502616 over 6 hops to deliver 1500000
2018-08-10T12:37:16.371Z lightningd(5892): 0250373555232cec757ea141273e75381c84cc3ab22f1e3a3ca5a2dc55b1904c24 chan #28: htlc 11 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2018-08-10T12:37:16.372Z lightningd(5892): pay(0x2decc08): Try another route
2018-08-10T12:37:17.104Z lightningd(5892): pay(0x2decc08): sendpay via route: us -> 531808:300:1 (1505225msat, 182blk) -> 0250373555232cec757ea141273e75381c84cc3ab22f1e3a3ca5a2dc55b1904c24 -> 533604:1588:0 (1504210msat, 168blk) -> 02cdf83ef8e45908b1092125d25c68dcec7751ca8d39f557775cd842e5bc127469 -> 536081:990:0 (1503209msat, 24blk) -> 0229cf81c21bbd21c2a41a4ae645933b89bb6d9a5920ca90e41ba270666879adab -> 536081:1007:0 (1503209msat, 10blk) -> 0211887ec50c8e7bb4bdb7259281a2314a6bdb70729f52845937f868fa44ad3748
2018-08-10T12:37:17.107Z lightningd(5892): Sending 1505225 over 4 hops to deliver 1500000
2018-08-10T12:37:18.653Z lightningd(5892): 0250373555232cec757ea141273e75381c84cc3ab22f1e3a3ca5a2dc55b1904c24 chan #28: htlc 12 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2018-08-10T12:37:18.654Z lightningd(5892): pay(0x2decc08): Try another route
2018-08-10T12:37:19.450Z lightningd(5892): pay(0x2decc08): sendpay via route: us -> 532656:2413:1 (1503998msat, 312blk) -> 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028 -> 532130:1901:1 (1502997msat, 168blk) -> 02cdf83ef8e45908b1092125d25c68dcec7751ca8d39f557775cd842e5bc127469 -> 536081:990:0 (1501996msat, 24blk) -> 0229cf81c21bbd21c2a41a4ae645933b89bb6d9a5920ca90e41ba270666879adab -> 536081:1007:0 (1501996msat, 10blk) -> 0211887ec50c8e7bb4bdb7259281a2314a6bdb70729f52845937f868fa44ad3748
2018-08-10T12:37:19.453Z lightningd(5892): Sending 1503998 over 4 hops to deliver 1500000
2018-08-10T12:37:21.346Z lightningd(5892): 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028 chan #34: htlc 77 failed from 0th node with code 0x1007 (WIRE_TEMPORARY_CHANNEL_FAILURE)
2018-08-10T12:37:21.346Z lightningd(5892): pay(0x2decc08): Try another route
lightningd: Outstanding taken pointers: (nil)
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe
lightning_channeld: Writing out status 65522: Broken pipe

FYI: When my node crashes, "num_taken" in "taken_any(void)" in file ccan/ccan/take/take.c is always 1.

Made a payment to coinpanic.com today and my node crashed again. I found out it was always node
cosmicApotheosis 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028
that caused the crash.

I then closed that channel and made payments to 5 different websites and had no problems anymore.
So it´s looks like a very rare bug to me. Good luck!

UPDATE: Well it's actually still crashing some time after I made payments (but in a different way).
See: out_180820181808.txt and crash_180820181808.log for a typical example here

[cdecker]

The relevant crash log is this:

+25742.517892697 lightningd(19783):BROKEN: FATAL SIGNAL 6 (version v0.6-322-g700a0d7)
+25742.517925657 lightningd(19783):BROKEN: backtrace: common/daemon.c:42 (crashdump) 0x4307f6
+25742.517942662 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0x7fa5c77494af
+25742.517953838 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0x7fa5c7749428
+25742.517964239 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0x7fa5c774b029
+25742.517973932 lightningd(19783):BROKEN: backtrace: ccan/ccan/tal/tal.c:93 (call_error) 0x4772d4
+25742.517986220 lightningd(19783):BROKEN: backtrace: ccan/ccan/tal/tal.c:165 (check_bounds) 0x477491
+25742.517998108 lightningd(19783):BROKEN: backtrace: ccan/ccan/tal/tal.c:174 (to_tal_hdr) 0x4774d0
+25742.518009206 lightningd(19783):BROKEN: backtrace: ccan/ccan/tal/tal.c:472 (tal_free) 0x477df7
+25742.518021424 lightningd(19783):BROKEN: backtrace: lightningd/peer_htlcs.c:1035 (peer_sending_commitsig) 0x427f4c
+25742.518033983 lightningd(19783):BROKEN: backtrace: lightningd/channel_control.c:130 (channel_msg) 0x409a2e
+25742.518046021 lightningd(19783):BROKEN: backtrace: lightningd/subd.c:474 (sd_msg_read) 0x42b076
+25742.518057532 lightningd(19783):BROKEN: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x469fb2
+25742.518068898 lightningd(19783):BROKEN: backtrace: ccan/ccan/io/io.c:387 (do_plan) 0x46aa83
+25742.518079876 lightningd(19783):BROKEN: backtrace: ccan/ccan/io/io.c:397 (io_ready) 0x46aac1
+25742.518092172 lightningd(19783):BROKEN: backtrace: ccan/ccan/io/poll.c:310 (io_loop) 0x46c621
+25742.518103434 lightningd(19783):BROKEN: backtrace: lightningd/lightningd.c:455 (main) 0x414b50
+25742.518114587 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0x7fa5c773482f
+25742.518124109 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0x403618
+25742.518132742 lightningd(19783):BROKEN: backtrace: (null):0 ((null)) 0xffffffffffffffff```

Updated to version v0.6.1rc1 and my node doesn't crash anymore, even though I made many payments.

However, for a version just before that and almost identical, v0.6-412-ga311527, I created a channel with cosmicApotheosis 03a8334aba5660e241468e2f0deb2526bfd50d0e3fe808d882913e39094dc1a028 again and after a payment my node immediately crashed with the same error ( lightningd: Outstanding taken pointers: (nil) ). I closed my channel to cosmicApotheosis.

So, without a channel to cosmicApotheosis I experience no problems at all, but with a channel to cosmicApotheosis I'm not so sure.

[cdecker]

Ok, marking this as stale and closing in a few days if it doesn't manifest itself anymore.

To test it, I will create a new channel to cosmicApotheosis and make a few payments. To be honest I expect my node to crash again. I will report back in a few days..

Connected to cosmicApotheosis, did some payments and my node didn't crash. The issue can be closed indeed. I'm now using the Spark wallet and it looks very professional already. Great work guys!