Handle server closing connection after response body is sent

[JoshMock]

There are cases where Elasticsearch may do the following:

1. decide to not read the entire request body
2. return a response with an HTTP error code
3. close the connection

One example of this is, if Elasticsearch receives a _bulk request where the content-length is larger than it is configured to handle, it will stop reading the request body after the first chunk is sent, respond with a 413 Content Too Large, and close the request.

In this case, the transport would see this as an EPIPE error. The problem is that, when using HttpConnection (which many customers still do, as well as Kibana), it would see the response body was finished being sent, via the request end event, and stop listening to all events on the request, including error events, before the connection was closed by the server and the EPIPE error would be raised. So, that unhandled error would bubble up the stack, and the transport would attempt to finish sending the request body over a now-closed connection.

HttpConnection was being too optimistic about what signals the end of a request/response cycle. When using the Node.js http library, as HttpConnection does, the following conditions are a better way to ensure that the cycle has finished:

• request finish and response end events have both fired (success)
• request error event has fired, signaling a connection-related issue (failure)
• response close event has fired before either of the above conditions are met, signaling an early connection closure (failure)

This fixes elastic/elasticsearch-js#2605.

[github-actions]

The backport to 8.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-8.x 8.x
# Navigate to the new working tree
cd .worktrees/backport-8.x
# Create a new branch
git switch --create backport-223-to-8.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b44dd134cc9bf1fbca5204851ec38030cdfd06e9
# Push it to GitHub
git push --set-upstream origin backport-223-to-8.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-8.x

Then, create a pull request where the base branch is 8.x and the compare/head branch is backport-223-to-8.x.

[pmuellr]

I can see the onResponseClose() is throwing Response aborted while reading the body, which is also emitted on an EPIPE (shown below in line 255). These two messages should be distinct, since we won't know which one caused this message.

It also appears we don't remove this listener. It seems like we should remove it in the same places we remove data and end, most specifically on onEnd(). I'm wondering if the listener is going off when the socket closes, making it seem like an otherwise successful call failed.

[JoshMock]

They're both EPIPE errors, which is why I gave them the same error message. It's the same symptom, just happening under slightly different circumstances: one gets a response body back, the other doesn't. That difference requires you to inspect the error's meta property to tell the difference, but I can use a slightly different message to help distinguish them further.

While the 'close' event should be implicitly cleaned up during garbage collection, it won't hurt to explicitly remove those listeners at the end of the lifecycle.

I've made both of these changes in #266. Please take a look!