HttpRequestException Lacks StatusCode Information on ProxyTunnelError

[cenceg]

Description

When HttpRequestException is thrown with ProxyTunnelError HttpRequestError, the StatusCode is not included. This behavior may not be ideal for scenarios involving web scraping where backconnecting proxy is utilized extensively.

In web scraping, we mostly do not connect to the proxy directly. Instead, a backconnecting proxy service is employed which dynamically assigns different proxies which is convenient to provide rotating proxies. The CONNECT request is critical as it is the point where the backconnecting proxy might reject a request. Typically, proxies communicate the nature of such rejections via HTTP status codes (e.g., 429 when request rate is too high, 502 when no available proxy). Some will use HTTP headers detailing the reason too.

Currently, we have to parse the Message to find the actual status code

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[wfurt]

do you have packet capture of such exchange @cenceg ?

[cenceg]

do you have packet capture of such exchange @cenceg ?

Sure, here's the curl of backconnecting proxy response stating no desired proxy is available