simple_http: actually reuse sockets

[darosior]

serde_json::from_reader requires that the other side close the socket for it to complete. Therefore we were forcing the server to always close streams to us, never reusing any single sockets. In fact, it was more a burden because after the first request for every request we would have to call the request() function twice: once for it to notice that the socket was closed on the other side (receiving an EOF on line 202 and confusingly returning an error telling the HTTP header was malformed).

Instead, get back to what was previously used in previous versions: read a line from the stream (using '\n' to mark the end of the request instead of a connection close) and parse it afterward.

From https://docs.rs/serde_json/latest/serde_json/fn.from_reader.html:

It is expected that the input stream ends after the deserialized object. If the stream does not end, such as in the case of a persistent socket connection, this function will not return. It is possible instead to deserialize from a prefix of an input stream without looking for EOF by managing your own Deserializer.

[darosior]

Maybe the fact we now have to make every request twice is (part of) the reason for the performance hit mentioned in #76 (comment)?

At least it can't be that we never actually reusing sockets, since we weren't able to in the first place.

[apoelstra]

@darosior yes, from_reader expects the stream of bytes to end. this is why we call take on the socket. There is no requirement that the other side closes the socket.

[apoelstra]

This PR reverts #69.

[darosior]

Closing for now. I don't remember the context anymore, but i think #79 is related.

EDIT: if one is looking for more context i discussed this issue with Andrew on IRC. So maybe look for #bitcoin-rust logs on Libera around the date this PR was opened.