Getting Incomplete response with debug_traceTransaction

[screw]

I am running archive node and when I try to use python requests library (or curl from shell) I get IncompleteRead for some transactions

relevant python code:

headers = {'accept': 'application/json', 'content-type': 'application/json'}
payload = {"jsonrpc": "2.0", "method": "debug_traceTransaction",
                           "params": [tx.hex()], "id": 83}
result = requests.post('http://127.0.0.1:10010', headers=headers, json=payload)
json_output = result.json()

and the result:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 443, in _error_catcher
    yield
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 818, in read_chunked
    chunk = self._handle_chunk(amt)
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 771, in _handle_chunk
    returned_chunk = self._fp._safe_read(self.chunk_left)
  File "/usr/lib64/python3.9/http/client.py", line 628, in _safe_read
    raise IncompleteRead(b''.join(s), amt)
http.client.IncompleteRead: IncompleteRead(6 bytes read, 2042 more expected)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 816, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 623, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 844, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.9/site-packages/urllib3/response.py", line 460, in _error_catcher
    raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(6 bytes read, 2042 more expected)', IncompleteRead(6 bytes read, 2042 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/eth_tx_status/main.py", line 66, in <module>
    result = requests.post('http://127.0.0.1:10010', headers=headers, json=payload)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 745, in send
    r.content
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 899, in content
    self._content = b"".join(self.iter_content(CONTENT_CHUNK_SIZE)) or b""
  File "/usr/local/lib/python3.9/site-packages/requests/models.py", line 818, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(6 bytes read, 2042 more expected)', IncompleteRead(6 bytes read, 2042 more expected))

using curl:

curl -H "Content-Type: application/json"--data '{"jsonrpc":"2.0","method":"debug_traceTransaction","params":["0xaa91b9638522c8a7686fc97ef4e5bec00b74b860a7e426825a49460ed025edab"],"id":83}' --url 127.0.0.1:10010

and the result:

<ommiting part of the response>
...
:"0000000000000000000000000000000000000000000000000000000000090009","9cbd36e402b21614adfadf3a9bb80b88c8ea865f2a07128c87e2fbf4c217d49f":"0000000000000000000000000000000000000000000000000000000000090008","9cbd36ecurl: (18) transfer closed with outstanding read data remaining

[rjl493456442]

Are you sure that you are running the archive node like gcmode=archive? If it's just a full-node, it might need a couple of minutes to recover the tracing state.

[screw]

Yes, I am sure I have gcmode archive set and run the node from scratch.
Am I correct to assume that this Enabling recording of key preimages since archive mode is used indicates that geth is indeed running in archive mode?

I can get every other transaction instantly.

Using geth console directly:

debug.traceTransaction("0xaa91b9638522c8a7686fc97ef4e5bec00b74b860a7e426825a49460ed025edab")
Error: Post "http://localhost:10010": EOF
        at web3.js:6365:9(45)
        at send (web3.js:5099:62(34))
        at <eval>:1:23(4)

and some random transaction a few blocks after the mentioned one

debug.traceTransaction("0xcd1120694dc742c5c4f65466f7ee8667e855d7331bc09c12ef728a4343c26b9f")
{
  failed: false,
  gas: 21000,
  returnValue: "",
  structLogs: []
}

and another transaction few blocks before

debug.traceTransaction("0xf0252fdd52a8569400ef6b899a0ad99b0312a33ae645751a286cd4af8b047c07")
{
  failed: false,
  gas: 21000,
  returnValue: "",
  structLogs: []
}

[holiman]

That tx burnt through 2.8M gas, back in 2015 (when a lot of now-expensive ops were cheap), it's probably a long / large trace, and sending it times out. In that curl-example, how long does it take before it errors?

[holiman]

You can try this instead:

> debug.standardTraceBlockToFile("0xff47a55638c883bd7049c034e0d3a8a8be5f0e8b6dda15770a24bc0082e4f88c",{"txHash": "0xaa91b9638522c8a7686fc97ef4e5bec00b74b860a7e426825a49460ed025edab"})

It should dump it out to disk, which is less resource-intense than storing it all in mem and then send over rpc.

[screw]

That tx burnt through 2.8M gas, back in 2015 (when a lot of now-expensive ops were cheap), it's probably a long / large trace, and sending it times out. In that curl-example, how long does it take before it errors?

geth console takes about 35s

same result with curl ~34s

the debug.standardTraceBlockToFile does work, just that I will have to implement the special case to handle these errors.

thank you.

[holiman]

the debug.standardTraceBlockToFile does work, just that I will have to implement the special case to handle these errors.

I was more wondering -- if you do that, what is the size of the result?

[screw]

here is the size of the result file

du /tmp/block_0xff47a556-0-0xaa91b963-680803741 
127892  /tmp/block_0xff47a556-0-0xaa91b963-680803741

Since it hovers around 30s I assume that is the default timeout for the HTTP response.
Is there a way to configure it?

[s1na]

Fixed via #25457