ambiguous `ChannelClosed` errors due to a race in `hyper-util`

[dare3path]

When using reqwest (v0.12) with hyper-util’s legacy HTTP client (backed by hyper v1.6.0), connection errors during HTTP/1.1 connection setup (e.g., TLS handshake failures, unexpected server responses) are sometimes masked as vague hyper::Error(ChannelClosed) errors. This occurs due to a race condition in hyper-util’s connection handling, where errors from the background connection task are discarded if the connection channel closes before readiness is confirmed. This makes debugging challenging, especially in scenarios like mTLS setups or servers sending unsolicited responses.

code to reproduce the issue (click me to expand)

Generate certs:
openssl req -x509 -newkey rsa:2048 -nodes -days 365 -keyout server.key -out server.crt -subj "/CN=localhost"
now you have server.crt and server.key in current dir.

Run this mtls_server.py python server which will serve errors(so to speak):

#!/usr/bin/python3
import socket
import ssl
import time

HOST = "127.0.0.1"
PORT = 8443
CERT_FILE = "server.crt"
KEY_FILE = "server.key"
CLIENT_CA_FILE = "client_ca.crt"

def create_ssl_context():
    context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    context.minimum_version = ssl.TLSVersion.TLSv1_3
    context.maximum_version = ssl.TLSVersion.TLSv1_3
    context.load_cert_chain(certfile=CERT_FILE, keyfile=KEY_FILE)
    context.verify_mode = ssl.CERT_REQUIRED
    context.load_verify_locations(cafile=CLIENT_CA_FILE)
    return context

def main():
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind((HOST, PORT))
    server_socket.listen(1)
    
    print(f"Server started on {HOST}:{PORT} with mTLS")
    
    context = create_ssl_context()
    ssl_server_socket = context.wrap_socket(server_socket, server_side=True, do_handshake_on_connect=True)
    
    while True:
        try:
            client_socket, addr = ssl_server_socket.accept()
            print(f"(unreachable) Connection from {addr}")
            
        except ssl.SSLError as e:
            print(f"TLS handshake error: {e}")
        except Exception as e:
            print(f"Error: {e}")
            continue
    
    ssl_server_socket.close()

if __name__ == "__main__":
    main()

Make a rust project (cargo new mtls_test && cd mtls_test)
this will be a TLS client that doesn't give client certs,
and replace the two files:
Cargo.toml

[package]
name = "mtls_test"
version = "0.1.0"
edition = "2024"

[dependencies]
reqwest = { version = "0.12", features = ["default", "native-tls"] }
tokio = { version = "1.44", features = ["full"] }
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }

[profile.dev]
debug=1

[patch.crates-io]
# Use a patched hyper-util crate/repo like this:
# cd /tmp
# git clone https://github.com/hyperium/hyper-util.git
# make sure the PR or patch is applied in /tmp/hyper-util
hyper-util = { path = "/tmp/hyper-util" }

src/main.rs:

use reqwest;
use std::time::Duration;
use tokio::time::sleep;
use reqwest::Certificate;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    // Initialize tracing, to see it do:
    // $ export RUST_LOG=trace
    tracing_subscriber::fmt()
        .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
        .init();


    // Send requests to observe error consistency
    for i in 0..20 {
    // Create reqwest client with TLS but no client certificate
    let client = reqwest::Client::builder()
        .timeout(Duration::from_millis(500))
        .danger_accept_invalid_certs(false)
        .add_root_certificate(
            Certificate::from_pem(
                &std::fs::read("../server.crt")?)?)
        .build()?;
        println!("-------- Sending request {}", i + 1);
        match client.get("https://localhost:8443/").send().await {
            Ok(response) => {
                let text = response.text().await?;
                println!("Received: {}", text);
            }
            Err(e) => {
                // Check if the error is the expected TLS "certificate required" error
                if format!("{:?}", e).contains("tlsv13 alert certificate required") {
                    println!("Ok");
                } else {
                    println!("{:?}", e);
                }
            }
        }
        //sleep(Duration::from_millis(500)).await;
    }

    println!("Waiting 1 second before exiting");
    sleep(Duration::from_secs(1)).await;

    Ok(())
}

outputs (click me to expand):

cargo run:

• without patch, outputs racy and wrong error(s):

-------- Sending request 1
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 2
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 3
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 4
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 5
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 6
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 7
Ok
-------- Sending request 8
Ok
-------- Sending request 9
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 10
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 11
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 12
Ok
-------- Sending request 13
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 14
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 15
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 16
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 17
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 18
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 19
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }
-------- Sending request 20
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }

• with patch, outputs:

     Running `target/debug/mtls_test`
-------- Sending request 1
Ok
-------- Sending request 2
Ok
-------- Sending request 3
Ok
-------- Sending request 4
Ok
-------- Sending request 5
Ok
-------- Sending request 6
Ok
-------- Sending request 7
Ok
-------- Sending request 8
Ok
-------- Sending request 9
Ok
-------- Sending request 10
Ok
-------- Sending request 11
Ok
-------- Sending request 12
Ok
-------- Sending request 13
Ok
-------- Sending request 14
Ok
-------- Sending request 15
Ok
-------- Sending request 16
Ok
-------- Sending request 17
Ok
-------- Sending request 18
Ok
-------- Sending request 19
Ok
-------- Sending request 20
Ok
Waiting 1 second before exiting

Those Ok are a placeholder for the correct error which is this:
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(Io, Custom { kind: Other, error: Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 167773276, library: "SSL routines", function: "ssl3_read_bytes", reason: "tlsv13 alert certificate required", file: "ssl/record/rec_layer_s3.c", line: 911, data: "SSL alert number 116" }]))) } })) }

instead of the wrong error which is:
reqwest::Error { kind: Request, url: "https://localhost:8443/", source: hyper_util::client::legacy::Error(SendRequest, hyper::Error(ChannelClosed)) }

re #1808 (comment)

PR is here hyperium/hyper-util#184

Thanks to Grok 3¹ (created by xAI) for helping me debug the ChannelClosed issue, develop this patch, and test it with reqwest and custom clients.

Footnotes

1. It literally would've been impossible without an AI/Grok3 because I would've given up long ago, heck I didn't even know what futures are and all that polling and async (not that I know now, but it's better than nothing). ↩

[dare3path]

Here's another set of repro. steps which shows that the error hyper::Error(UnexpectedMessage) due to DEBUG hyper::proto::h1::conn: received unexpected 97 bytes on an idle connection happens to be hidden/overwritten by hyper::Error(ChannelClosed) (if without patch) which is what happens when a non-http compliant server(like echo $reply | nc -l) sends the reply without first getting a request from the client, emulating a netcat connection.

We have here 3 blocking requests, then 3 async ones, then 3 blocking ones again. Only the 3 async ones fail and it matter with what error they fail.

Output when not patched: (click me to expand)

     Running `target/debug/requester_tokio`
spawned
main, after waited
2025-04-19T01:01:58.989733Z TRACE reqwest::blocking::wait: (ThreadId(8)) park without timeout    
2025-04-19T01:01:58.990763Z TRACE reqwest::blocking::client: (ThreadId(9)) start runtime::block_on    
---------- req: 1 
2025-04-19T01:01:59.138379Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:01:59.138453Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.991335ms    
2025-04-19T01:01:59.139152Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:01:59.139229Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:01:59.139229Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:01:59.142426Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:01:59.142837Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:01:59.143115Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:01:59.143300Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:01:59.143648Z TRACE reqwest::connect::verbose: ff862afd write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:01:59.143913Z TRACE reqwest::connect::verbose: ff862afd read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:01:59.144749Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 2 
2025-04-19T01:02:00.146732Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:02:00.146858Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.979037ms    
2025-04-19T01:02:00.147434Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:00.147815Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:00.148432Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:00.152854Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:00.154335Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:00.154835Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:00.155506Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:02:00.156469Z TRACE reqwest::connect::verbose: 7d78d4e8 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:02:00.159577Z TRACE reqwest::connect::verbose: 7d78d4e8 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:00.161346Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 3 
2025-04-19T01:02:01.164354Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:02:01.164466Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:01.164526Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.976242ms    
2025-04-19T01:02:01.164614Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:01.164727Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:01.169237Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:01.177142Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:01.177503Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:01.177793Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:02:01.178476Z TRACE reqwest::connect::verbose: 61361533 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:02:01.178981Z TRACE reqwest::connect::verbose: 61361533 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:01.182140Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
2025-04-19T01:02:01.182379Z TRACE reqwest::blocking::client: closing runtime thread (ThreadId(9))    
2025-04-19T01:02:01.182450Z TRACE reqwest::blocking::client: signaled close for runtime thread (ThreadId(9))    
2025-04-19T01:02:01.182514Z TRACE reqwest::blocking::client: (ThreadId(9)) Receiver is shutdown    
2025-04-19T01:02:01.183899Z TRACE reqwest::blocking::client: (ThreadId(9)) end runtime::block_on    
2025-04-19T01:02:01.184388Z TRACE reqwest::blocking::client: (ThreadId(9)) finished    
2025-04-19T01:02:01.185057Z TRACE reqwest::blocking::client: closed runtime thread (ThreadId(9))    
========== Final (blocking) request, response: Hello World!
---------- async req 1
2025-04-19T01:02:02.222310Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:02.222412Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:02.222477Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:02.223132Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:02.223362Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:02.223450Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:02.223630Z TRACE reqwest::connect::verbose: 1aa41b20 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:02.223747Z DEBUG hyper_util::client::legacy::client: client connection error: received unexpected message from connection
2025-04-19T01:02:02.223746Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 1/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            ChannelClosed,
        ),
    ),
)
---------- async req 2
2025-04-19T01:02:03.229449Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:03.229491Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:03.229522Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:03.230144Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:03.230555Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:03.230637Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:03.230811Z TRACE reqwest::connect::verbose: 899cd7d1 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:03.230896Z DEBUG hyper_util::client::legacy::client: client connection error: received unexpected message from connection
2025-04-19T01:02:03.230935Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 2/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            ChannelClosed,
        ),
    ),
)
---------- async req 3
2025-04-19T01:02:04.234390Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:04.234451Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:04.234525Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:04.235427Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:04.235838Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:04.236002Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:04.236219Z TRACE reqwest::connect::verbose: 6fb37fe3 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:04.236414Z DEBUG hyper_util::client::legacy::client: client connection error: received unexpected message from connection
2025-04-19T01:02:04.236444Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 3/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            ChannelClosed,
        ),
    ),
)
Giving up after 3 attempts
Final (async) request, respone: <failed>
2025-04-19T01:02:05.241143Z TRACE reqwest::blocking::wait: (ThreadId(8)) park without timeout    
2025-04-19T01:02:05.241289Z TRACE reqwest::blocking::client: (ThreadId(11)) start runtime::block_on    
---------- req: 1 
2025-04-19T01:02:05.366155Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:02:05.366193Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.994969ms    
2025-04-19T01:02:05.366542Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:05.366613Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:05.366641Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:05.367423Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:05.367611Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:05.367704Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:05.367787Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:02:05.367920Z TRACE reqwest::connect::verbose: 92e00c10 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:02:05.368119Z TRACE reqwest::connect::verbose: 92e00c10 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:05.368426Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 2 
2025-04-19T01:02:06.371452Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:02:06.371790Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:06.371878Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.972049ms    
2025-04-19T01:02:06.379618Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:06.381269Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:06.384737Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:06.385849Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:06.386505Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:06.386939Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:02:06.388961Z TRACE reqwest::connect::verbose: 4c79abc5 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:02:06.390367Z TRACE reqwest::connect::verbose: 4c79abc5 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:06.390701Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 3 
2025-04-19T01:02:07.402431Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:02:07.402855Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.97177ms    
2025-04-19T01:02:07.402467Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:02:07.403441Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:02:07.403806Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:02:07.407444Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:02:07.407621Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:02:07.407708Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:02:07.407789Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:02:07.407919Z TRACE reqwest::connect::verbose: 548c3c82 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:02:07.408068Z TRACE reqwest::connect::verbose: 548c3c82 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:02:07.408395Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
2025-04-19T01:02:07.408557Z TRACE reqwest::blocking::client: closing runtime thread (ThreadId(11))    
2025-04-19T01:02:07.408641Z TRACE reqwest::blocking::client: signaled close for runtime thread (ThreadId(11))    
2025-04-19T01:02:07.408689Z TRACE reqwest::blocking::client: (ThreadId(11)) Receiver is shutdown    
2025-04-19T01:02:07.409505Z TRACE reqwest::blocking::client: (ThreadId(11)) end runtime::block_on    
2025-04-19T01:02:07.409687Z TRACE reqwest::blocking::client: (ThreadId(11)) finished    
2025-04-19T01:02:07.410039Z TRACE reqwest::blocking::client: closed runtime thread (ThreadId(11))    
========== Final (blocking) request, response: Hello World!
returned

You saw those 3 ChannelClosed in the middle, for the 3 async requests. This is incorrect due to the issue.

Output when patched: (click me to expand)

     Running `target/debug/requester_tokio`
spawned
main, after waited
2025-04-19T01:04:14.533512Z TRACE reqwest::blocking::wait: (ThreadId(8)) park without timeout    
2025-04-19T01:04:14.533942Z TRACE reqwest::blocking::client: (ThreadId(9)) start runtime::block_on    
---------- req: 1 
2025-04-19T01:04:14.678560Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:14.679883Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.99441ms    
2025-04-19T01:04:14.680089Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:14.680194Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:14.680262Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:14.681415Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:14.681542Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:14.681580Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:14.681597Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:14.681628Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:14.681649Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:14.681739Z TRACE reqwest::connect::verbose: 1e5a3ba2 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:14.681788Z TRACE reqwest::connect::verbose: 1e5a3ba2 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:14.681986Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 2 
2025-04-19T01:04:15.684757Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:15.685468Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:15.685595Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.972329ms    
2025-04-19T01:04:15.685676Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:15.685836Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:15.689349Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:15.689401Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:15.689766Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:15.689800Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:15.689840Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:15.689862Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:15.690019Z TRACE reqwest::connect::verbose: e7cc881c write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:15.690118Z TRACE reqwest::connect::verbose: e7cc881c read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:15.690391Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 3 
2025-04-19T01:04:16.756929Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:16.757412Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:16.757433Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.972329ms    
2025-04-19T01:04:16.757721Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:16.757880Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:16.760147Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:16.760277Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:16.760314Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:16.760330Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:16.760360Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:16.760377Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:16.760465Z TRACE reqwest::connect::verbose: 4605ffc8 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:16.760515Z TRACE reqwest::connect::verbose: 4605ffc8 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:16.760723Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
2025-04-19T01:04:16.760866Z TRACE reqwest::blocking::client: closing runtime thread (ThreadId(9))    
2025-04-19T01:04:16.760921Z TRACE reqwest::blocking::client: signaled close for runtime thread (ThreadId(9))    
2025-04-19T01:04:16.761084Z TRACE reqwest::blocking::client: (ThreadId(9)) Receiver is shutdown    
2025-04-19T01:04:16.761782Z TRACE reqwest::blocking::client: (ThreadId(9)) end runtime::block_on    
2025-04-19T01:04:16.762062Z TRACE reqwest::blocking::client: (ThreadId(9)) finished    
2025-04-19T01:04:16.762422Z TRACE reqwest::blocking::client: closed runtime thread (ThreadId(9))    
========== Final (blocking) request, response: Hello World!
---------- async req 1
2025-04-19T01:04:17.798081Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:17.798139Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:17.798173Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:17.798720Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:17.799033Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:17.799124Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:17.799193Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:17.799313Z TRACE reqwest::connect::verbose: 3f2bf236 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:17.799436Z DEBUG hyper_util::client::legacy::client: client connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:17.799449Z TRACE hyper_util::client::legacy::client: sending connection error to error channel
2025-04-19T01:04:17.799454Z TRACE hyper_util::client::legacy::client: connection channel closed, checking for connection error
2025-04-19T01:04:17.799472Z TRACE hyper_util::client::legacy::client: received connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:17.799491Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 1/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            UnexpectedMessage,
        ),
    ),
)
---------- async req 2
2025-04-19T01:04:18.810775Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:18.810964Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:18.811691Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:18.815395Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:18.817717Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:18.825691Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:18.826570Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:18.829336Z TRACE reqwest::connect::verbose: 98f19688 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:18.830115Z DEBUG hyper_util::client::legacy::client: client connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:18.830401Z TRACE hyper_util::client::legacy::client: sending connection error to error channel
2025-04-19T01:04:18.830650Z TRACE hyper_util::client::legacy::client: connection channel closed, checking for connection error
2025-04-19T01:04:18.831206Z TRACE hyper_util::client::legacy::client: received connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:18.831348Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 2/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            UnexpectedMessage,
        ),
    ),
)
---------- async req 3
2025-04-19T01:04:19.835830Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:19.836497Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:19.837242Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:19.840393Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:19.840802Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:19.840849Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:19.840881Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:19.841048Z TRACE reqwest::connect::verbose: 1a021563 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:19.841147Z DEBUG hyper_util::client::legacy::client: client connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:19.841163Z TRACE hyper_util::client::legacy::client: connection channel closed, checking for connection error
2025-04-19T01:04:19.841178Z TRACE hyper_util::client::legacy::client: sending connection error to error channel
2025-04-19T01:04:19.841240Z TRACE hyper_util::client::legacy::client: received connection error: hyper::Error(UnexpectedMessage)
2025-04-19T01:04:19.841291Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
!!!!!!!!!! Request attempt 3/3 failed: error sending request for url (http://localhost:8080/), Some(
    hyper_util::client::legacy::Error(
        SendRequest,
        hyper::Error(
            UnexpectedMessage,
        ),
    ),
)
Giving up after 3 attempts
Final (async) request, respone: <failed>
2025-04-19T01:04:20.844334Z TRACE reqwest::blocking::wait: (ThreadId(8)) park without timeout    
2025-04-19T01:04:20.844484Z TRACE reqwest::blocking::client: (ThreadId(11)) start runtime::block_on    
---------- req: 1 
2025-04-19T01:04:20.969934Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:20.970508Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.981832ms    
2025-04-19T01:04:20.972546Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:20.972773Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:20.972874Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:20.975562Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:20.976162Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:20.976576Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:20.976756Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:20.976814Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:20.977268Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:20.977714Z TRACE reqwest::connect::verbose: 2e22dd65 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:20.978308Z TRACE reqwest::connect::verbose: 2e22dd65 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:20.979404Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 2 
2025-04-19T01:04:21.980039Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:21.980128Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.993851ms    
2025-04-19T01:04:21.980240Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:21.980280Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:21.980316Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:21.981009Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:21.981172Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:21.981249Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:21.981291Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:21.981347Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:21.981390Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:21.981475Z TRACE reqwest::connect::verbose: 4c0cdfe9 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:21.981520Z TRACE reqwest::connect::verbose: 4c0cdfe9 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:21.981731Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
---------- req: 3 
2025-04-19T01:04:22.993774Z TRACE reqwest::blocking::wait: wait at most 500ms    
2025-04-19T01:04:22.995310Z TRACE reqwest::blocking::wait: (ThreadId(8)) park timeout 499.512824ms    
2025-04-19T01:04:22.998277Z TRACE hyper_util::client::legacy::pool: checkout waiting for idle connection: ("http", localhost:8080)
2025-04-19T01:04:22.998636Z DEBUG reqwest::connect: starting new connection: http://localhost:8080/    
2025-04-19T01:04:22.999142Z TRACE hyper_util::client::legacy::connect::http: Http::connect; scheme=Some("http"), host=Some("localhost"), port=Some(Port(8080))
2025-04-19T01:04:23.003878Z DEBUG hyper_util::client::legacy::connect::http: connecting to 127.0.0.1:8080
2025-04-19T01:04:23.005280Z DEBUG hyper_util::client::legacy::connect::http: connected to 127.0.0.1:8080
2025-04-19T01:04:23.005741Z TRACE hyper_util::client::legacy::client: http1 handshake complete, spawning background dispatcher task
2025-04-19T01:04:23.006239Z TRACE hyper_util::client::legacy::client: waiting for connection to be ready
2025-04-19T01:04:23.006622Z TRACE hyper_util::client::legacy::client: connection is ready
2025-04-19T01:04:23.006928Z TRACE hyper_util::client::legacy::pool: checkout dropped for ("http", localhost:8080)
2025-04-19T01:04:23.009476Z TRACE reqwest::connect::verbose: 528f1e20 write (vectored): b"GET / HTTP/1.1\r\naccept: */*\r\nhost: localhost:8080\r\n\r\n"    
2025-04-19T01:04:23.010220Z TRACE reqwest::connect::verbose: 528f1e20 read: b"HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!\n"    
2025-04-19T01:04:23.014922Z TRACE reqwest::blocking::wait: wait at most 500ms    
Got 'Hello World!'
2025-04-19T01:04:23.021558Z TRACE reqwest::blocking::client: closing runtime thread (ThreadId(11))    
2025-04-19T01:04:23.021751Z TRACE reqwest::blocking::client: signaled close for runtime thread (ThreadId(11))    
2025-04-19T01:04:23.021908Z TRACE reqwest::blocking::client: (ThreadId(11)) Receiver is shutdown    
2025-04-19T01:04:23.022892Z TRACE reqwest::blocking::client: (ThreadId(11)) end runtime::block_on    
2025-04-19T01:04:23.023409Z TRACE reqwest::blocking::client: (ThreadId(11)) finished    
2025-04-19T01:04:23.024102Z TRACE reqwest::blocking::client: closed runtime thread (ThreadId(11))    
========== Final (blocking) request, response: Hello World!
returned

You saw those 3 UnexpectedMessage in the middle. This is correct, or intended behavior if the issue wasn't present.

The code used for the above is this:

Details

Cargo.toml

[package]
name = "requester_tokio"
version = "0.1.0"
edition = "2024"

[dependencies]
reqwest={ version="0.12.15", features=["blocking"] }
tokio={ version="1.44.2", features=["full"] }
http = "1.1"
tracing = "0.1"
tracing-subscriber = { version = "0.3.19", features = ["env-filter"] }

[profile.dev]
debug=1
debug-assertions = true
overflow-checks = true
codegen-units = 256
opt-level=0

[patch.crates-io]
hyper-util = { path = "/home/user/SOURCE/my/channelclosed.reqwest/hyper-util" }

src/main.rs:

use reqwest;
use tokio;
use std::error::Error;
use std::io;
//use reqwest::Client;
//use tokio::process::Command;
//use std::process::Output;
//use tokio::io::AsyncWriteExt;
use std::io::Write;
use std::time::Duration;


use tokio::task::JoinHandle;

fn flush() { //-> Result<(), Box<dyn std::error::Error>> {
    std::io::stderr().flush().unwrap();
    std::io::stdout().flush().unwrap();
//    tokio::io::stderr().flush().await?;
//    tokio::io::stdout().flush().await?;
//    Ok(())
}
async fn run_blocking() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    // Run blocking reqwest calls in a separate thread
    let blocking_result = tokio::task::spawn_blocking(move || {
        // Create a blocking Reqwest client
        let bclient = reqwest::blocking::Client::builder()
            .tcp_keepalive(Duration::from_secs(30))
            .connection_verbose(true)
            .timeout(Duration::from_millis(500)) // Quick timeout to fail fast
            .build()?;

        std::thread::sleep(Duration::from_millis(100));

        // Retry the blocking request with delays
        let max_attempts = 3;
        let mut attempts = 0;
        let final_response = loop {
            attempts += 1;
            eprintln!("---------- req: {} ", attempts);
            match bclient.get("http://localhost:8080/").send() {
                Ok(response) => match response.text() {
                    Ok(text) => {
                        eprintln!("Got '{}'", text);
                        if attempts>=max_attempts {
                            break text;
                        }
                    },
                    Err(e) => eprintln!("!!!!!!!!!! Failed to read response text: {}", e),
                },
                Err(e) => {
                    eprintln!("!!!!!!! Request attempt {}/{} failed: {}", attempts, max_attempts, e);
                    if attempts >= max_attempts {
                        eprintln!("Giving up after {} attempts", max_attempts);
                        return Err(Box::new(io::Error::new(
                            io::ErrorKind::Other,
                            format!("!!! Failed to connect after {} attempts: {}", max_attempts, e),
                        )) as Box<dyn std::error::Error + Send + Sync>);
                    }
                }
            }
            std::thread::sleep(Duration::from_millis(1000));
        };

        Ok(final_response)
    });

    // Explicitly handle the spawn_blocking result
    let blocking_result = match blocking_result.await {
        Ok(Ok(result)) => result,
        Ok(Err(e)) => return Err(e),
        Err(e) => return Err(Box::new(e) as Box<dyn std::error::Error + Send + Sync>),
    };

    println!("========== Final (blocking) request, response: {}", blocking_result);
    Ok(())
}

#[tokio::main]
//async fn main() {
//async fn main() -> Result<(), Box<dyn std::error::Error>> {
async fn main() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
    //env_logger::init();
    use tracing_subscriber;
tracing_subscriber::fmt()
    //.with_env_filter("hyper=trace,reqwest=trace,h2=trace")
    .with_env_filter("trace")
    .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
    .init();
//    // Print the current working directory
//    match std::env::current_dir() {
//        Ok(cwd) => eprintln!("Current working directory: {}", cwd.display()),
//        Err(e) => eprintln!("Failed to get current working directory: {}", e),
//    }

    let handle: JoinHandle<Result<_, _>> = tokio::spawn(async {
    //tokio::spawn(
        eprintln!("spawned");
        let cwd = std::env::current_dir()?;
        // script is in ../server.sh not in current dir, assuming cwd is project dir!
        let script_path = cwd.join("..").join("server.sh");
//        let script_path = cwd.join("..").join("server.py");
        let output = tokio::process::Command::new(script_path)
            .kill_on_drop(true)
            .output().await;
        eprintln!("returned");
        if let Err(ref e) = output {
            eprintln!("Server error: {}", e);
        }
        flush();
        output
//    ,);
    });
    // Allow some time for the server to start
    tokio::time::sleep(std::time::Duration::from_millis(1000)).await;
    eprintln!("main, after waited");
    flush();


    run_blocking().await?;
    tokio::time::sleep(std::time::Duration::from_millis(1000)).await;
    // Create a Reqwest client using ClientBuilder
    let client = reqwest::Client::builder()
        .tcp_keepalive(std::time::Duration::from_secs(30))
        //.pool_max_idle_per_host(0) //no effect, https://github.com/getsentry/uptime-checker/pull/204
        .pool_max_idle_per_host(1000)
        .connection_verbose(true)
        .http1_only()
        .timeout(Duration::from_millis(500))
//        .no_keep_alive() //not a thing
        .build()?;
//    // Replace the original reqwest::get with client.get
//    //XXX: good:
//    let final_response = client
//        .get("http://localhost:8080/")
//        .send()
//        .await?
//        .text()
//        .await?;
    // Retry the final request with delays
    let max_attempts = 3;
    let mut attempts = 0;
    let final_response = loop {
        attempts += 1;
        println!("---------- async req {}", attempts);
        match client.get("http://localhost:8080/").send().await {
            Ok(response) => {
                match response.text().await {
                    Ok(text) => {
                        eprintln!("+++++++++++++ Got '{}'", text);
                        if attempts>=max_attempts {
                            break text;
                        }
                    },
//                    Ok(text) => {
//                        break text
//                    }, // Success, exit loop with the response text
                    Err(e) => eprintln!("!!!!!!!!!!!!! Failed to read response text: {}", e),
                }
            }
            Err(e) => {
                eprintln!("!!!!!!!!!! Request attempt {}/{} failed: {}, {:#?}", attempts, max_attempts, e, e.source());
                if attempts >= max_attempts {
                    eprintln!("Giving up after {} attempts", max_attempts);
                    break "<failed>".to_string();//Err(format!("Failed to connect after {} attempts: {}", max_attempts, e).into());
                }
            }
        }
        // Wait before retrying
        tokio::time::sleep(Duration::from_millis(1000)).await;
    };
    println!("Final (async) request, respone: {}", final_response);

    tokio::time::sleep(std::time::Duration::from_millis(1000)).await;
    run_blocking().await?;

    //reqwest::get("http://localhost:8080/").await.unwrap();
    // Check the result of the spawned task
    match handle.await? {
        Ok(output) => {
            if !output.status.success() {
                eprintln!("Server failed with status: {}", output.status);
                if !output.stderr.is_empty() {
                    eprintln!("Server stderr: {}", String::from_utf8_lossy(&output.stderr));
                }
            }
        }
        Err(e) => eprintln!("Server error: {}", e),
    }
    tokio::time::sleep(std::time::Duration::from_millis(500)).await;
    Ok(())
}

../server.sh is:

#!/bin/sh
hello() {
	echo -e "HTTP/1.1 200 OK\r\nConnection: Close\r\nContent-Length: 12\r\nContent-Type: text/plain\r\n\r\nHello World!"
}
let 'count=0'
while true; do
	#hello | busybox nc -l -p 8080
  echo "started $0"
  #XXX: this sends the reply before it receives the GET request!
	hello | nc -l -p 8080
  echo "done"
  let 'count=count+1'
  sleep 0.2
  if test "$count" -ge "9"; then
    break
  fi
done

this is inspired from OP of #1808

run like:

export RUST_LOG=trace
cargo run

(you don't have to run ../server.sh because the rust program runs it and it exits automatically after 9 requests so you won't have to C-c the rust program, unless you use ../server.py from below)

../server.py is:
(you don't need this unless you uncomment the proper line in the rust program, and comment out the .sh one)

#!/usr/bin/python3
import socket
import time

HOST = "127.0.0.1"
PORT = 8080

# Set to False to send response on connect(mimimcs netcat); True to require valid GET request
# XXX: set to True to get rid of the issue in OP aka https://github.com/seanmonstar/reqwest/issues/1808
# True makes it align with HTTP/1.1 and makes the async requests (in the rust client) succeed.
# run client with: $ cargo run --example client --features=full -- http://127.0.0.1:8080
# ^ in hyper's repo! Repeatedly run it to see different (racey) errors (add a delay below if hitting same error)
REQUIRE_GET = False
#REQUIRE_GET = True

# Delay in milliseconds before closing connection after response; 0 for immediate close
#DELAY_CLOSE_MS = 250
DELAY_CLOSE_MS = 0
#with delay 250, the rust program can TRACE hyper::proto::h1::conn: shut down IO complete
#with delay 0 tho, it yields: 
#2025-04-18T19:29:21.872159Z DEBUG hyper::proto::h1::conn: error shutting down IO: Transport endpoint is not connected (os error 107)
#!!! recv_msg: no callback or queued request, propagating error err=hyper::Error(Shutdown, Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" })
#2025-04-18T19:29:21.872204Z DEBUG hyper_util::client::legacy::client: client connection error: hyper::Error(Shutdown, Os { code: 107, kind: NotConnected, message: "Transport endpoint is not connected" })


#DELAY_SEND_MS = 600
DELAY_SEND_MS = 0

RESPONSE = (
    b"HTTP/1.1 200 OK\r\n"
    b"Connection: Close\r\n"
    b"Content-Length: 12\r\n"
    b"Content-Type: text/plain\r\n"
    b"\r\n"
    b"Hello World!\n"
)

def is_valid_get_request(data):
    """Check if the received data is a valid HTTP GET request for /."""
    #print(f"Raw request: {data!r}")  # Log raw bytes
    try:
        # Decode first line of request
        first_line = data.decode("utf-8").split("\r\n")[0]
        #print(f"First line: {first_line!r}")
        method, path, version = first_line.split(" ")
        #print(f"Parsed: method={method!r}, path={path!r}, version={version!r}")
        return (
            method == "GET"
            and path == "/"
            and version.startswith("HTTP/1.")
        )
    except (UnicodeDecodeError, ValueError, IndexError):
        print(f"Validation failed: {e}")
        return False

def main():
    # Create TCP socket
    server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_socket.bind((HOST, PORT))
    server_socket.listen(1)
    
    mode = "require GET request" if REQUIRE_GET else "send response on connect"
    print(f"Server started on {HOST}:{PORT} (mode: {mode})")
    
    while True:
        try:
            # Accept client connection
            client_socket, addr = server_socket.accept()
            print(f"Connection from {addr}")
            
            if REQUIRE_GET:
                # Receive request (up to 1024 bytes)
                data = client_socket.recv(1024)
                
                # Validate GET request
                if is_valid_get_request(data):
                    print("Received valid GET request, sending response")
                    client_socket.sendall(RESPONSE)
                else:
                    print("Received invalid request, closing connection")
                    bad_request = (
                        b"HTTP/1.1 400 Bad Request\r\n"
                        b"Connection: Close\r\n"
                        b"Content-Length: 0\r\n"
                        b"\r\n"
                    )
                    client_socket.sendall(bad_request)
            else:
                # Send response immediately on connect ?
                if DELAY_SEND_MS > 0:
                    print(f"Delaying send by {DELAY_SEND_MS}ms")
                    time.sleep(DELAY_SEND_MS / 1000.0)  # Convert ms to seconds
                print("Sending response on connect")
                client_socket.sendall(RESPONSE)
            
            # Close connection immediately or delay first:
            if DELAY_CLOSE_MS > 0:
                print(f"Delaying connection close by {DELAY_CLOSE_MS}ms")
                time.sleep(DELAY_CLOSE_MS / 1000.0)  # Convert ms to seconds

            client_socket.close()
            print("Connection closed")
            
            # Small delay to ensure client processes response
            time.sleep(0.2)
            
        except KeyboardInterrupt:
            print("\nShutting down server")
            break
        except Exception as e:
            print(f"Error: {e}")
            continue
    
    server_socket.close()

if __name__ == "__main__":
    main()

EDIT: Actually, you might get less tracing output unless you do additionally:

Details

export RUSTFLAGS='--cfg hyper_unstable_tracing' 

before cargo run
and you must have Cargo.toml in addition:

[patch.crates-io]
reqwest = { path = "/home/user/SOURCE/my/channelclosed.reqwest/reqwest" }
hyper = { path = "/home/user/SOURCE/my/channelclosed.reqwest/hyper" }

in addition to the hyper-util line,
otherwise you won't see something like this:

2025-04-19T01:25:59.380207Z TRACE hyper::proto::h1::io: received 97 bytes
2025-04-19T01:25:59.380254Z DEBUG hyper::proto::h1::conn: received unexpected 97 bytes on an idle connection

which is very telling.

In those paths you just git clone their respective repos, and change the paths to where you cloned them.

[seanmonstar]

hyper-util v0.1.13 has been released with the fix for this, thanks!