InactiveRpcError, StatusCode.UNAVAILABLE #4031
Description
Background
I have two instances of lnd running on the same machine. One making a regular payment to the other. I have a script that loops and is checking to see if an invoice has been paid (amount other things) to 127.0.0.1:10010, using LookupInvoice. The loop sleeps for 75ms every time, so the rpc call is happening less than that frequency. After about 10 minutes of looping and receiving payments, the receiver crashes with the following error.
Traceback (most recent call last):
File "wall.py", line 131, in <module>
OutstandingInvoiceStatus=lnd.lookup_invoice(OutstandingInvoice.r_hash)
File "/home/xxxx/.local/lib/python3.7/site-packages/lndgrpc/errors.py", line 30, in wrapper
raise exc
File "/home/xxxx/.local/lib/python3.7/site-packages/lndgrpc/errors.py", line 19, in wrapper
return fnc(*args, **kwargs)
File "/home/xxxx/.local/lib/python3.7/site-packages/lndgrpc/client.py", line 132, in lookup_invoice
response = self._ln_stub.LookupInvoice(request)
File "/home/xxxx/.local/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/home/xxxx/.local/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1582683263.546610256","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3941,"referenced_errors":[{"created":"@1582683263.546599150","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":393,"grpc_status":14}]}"
>
About 30% of the time, the sender (another script), also crashes at the same time with the same error, with a call of DecodePayReq to 127.0.0.1:10009 (and it loops at a similar frequency).
If you'll notice above, I'm using the module lndgrpc, which includes some convenience wrapper functions for the grpc interface. In it's source code, it indicates that with every request, a new secure_channel is created to the interface.
https://github.com/adrienemery/lnd-grpc-client/blob/master/lndgrpc/common.py#L103
I'm wondering if there is a limit to the number of secure_channels that can be created? Maybe I'm looping too fast and the existing secure_channels aren't timing out quick enough in lnd, so eventually it won't let any more be created? Is there a way to close out a secure_channel so that this does not happen? Is lndgrpc's approach a bad design?
Your environment
- version of
lnd: commit 38b521d - ubuntu 18.04
bitcoind