ProxyAgent has different behavior from curl -x

[joyeecheung]

Bug Description

When testing nodejs/node#57165 I noticed that there are some behavior differences between ProxyAgent and curl which may be interesting. Most notably, ProxyAgent always tunnels with CONNECT even for pure HTTP traffic.

Reproducible By

Create a proxy server, like this:

// server.js
'use strict';

const http = require('http');
const net = require('net');

function log(req) {
  console.log('----- Received Request -----');
  console.log(`${req.method} ${req.url} HTTP/${req.httpVersion}`);
  for (const header in req.headers) {
    console.log(`${header}: ${req.headers[header]}`);
  }
}

const server = http.createServer((req, res) => {
  log(req);
  const [hostname, port] = req.headers.host.split(':');
  const targetPort = port || 80;

  const options = {
    hostname: hostname,
    port: targetPort,
    path: req.url,
    method: req.method,
    headers: req.headers
  };

  const proxyReq = http.request(options, proxyRes => {
    res.writeHead(proxyRes.statusCode, proxyRes.headers);
    proxyRes.pipe(res, { end: true });
  });

  proxyReq.on('error', err => {
    console.error('Proxy request error: ' + err.message);
    res.writeHead(500);
    res.end('Proxy error: ' + err.message);
  });

  req.pipe(proxyReq, { end: true });
});

server.on('connect', function (req, clientSocket, head) {
  log(req);
  const [hostname, port] = req.url.split(':');

  const serverSocket = net.connect(port, hostname, () => {
    clientSocket.write(
      'HTTP/1.1 200 Connection Established\r\n' +
      'Proxy-agent: Node.js-Proxy\r\n' +
      '\r\n'
    );
    serverSocket.write(head);
    clientSocket.pipe(serverSocket);
    serverSocket.pipe(clientSocket);
  });

  serverSocket.on('error', (err) => {
    console.error('Error on CONNECT tunnel:', err.message);
    clientSocket.write('HTTP/1.1 500 Connection Error\r\n\r\n');
    clientSocket.end();
  });
});

server.listen(8000, '127.0.0.1', () => {
  const { address, port } = server.address();
  console.log(`Proxy server listening on http://${address}:${port}`);
});

And send a request to http://example.com like this:

// proxy
const { fetch, ProxyAgent } = require('undici');

(async () => {
  const res = await fetch('http://example.com', {
    dispatcher: new ProxyAgent('http://127.0.0.1:8000')
  });
  console.log(await res.text());
})();

Expected Behavior

If I run curl -x http://127.0.0.1:8000 http://example.com, this is the output from the proxy server:

GET http://example.com/ HTTP/1.1
host: example.com
user-agent: curl/8.7.1
accept: */*
proxy-connection: Keep-Alive

Logs & Screenshots

Using the client with undici, the proxy server always get a CONNECT:

CONNECT example.com:80 HTTP/1.1
host: example.com
connection: close

This is somewhat similar to the one I get for sending requests to https://example.com using curl - which does use CONNECT.

Output from curl -x http://127.0.0.1:8000 https://example.com

CONNECT example.com:443 HTTP/1.1
host: example.com:443
user-agent: curl/8.7.1
proxy-connection: Keep-Alive

Output from client with undici ProxyAgent (change the http://example.com to https://example.com in the snippet above)

CONNECT example.com:80 HTTP/1.1
host: example.com
connection: close

Environment

macOS though I am sure this is irrelevant.

Additional context

I am not sure this really counts as a bug or not. But from my impression, not all proxy servers in the wild supports tunneling on non-443 ports. It may be safer to follow what curl does, because the setup of proxy server and target servers are not always in the control of users. If they end up having to work with a proxy server that does not support tunnelling on non-443 ports, and also having to connect to a server on 80 port or others with the proxy, there would be no way for them to work around it.

[mcollina]

I agree we should follow what curl is doing, but... I'm not sure how impactful this is, as we shouldn't really encourage anyone to pass through a proxy with plain HTTP.

Anyhow, yes, this looks like a bug.

This behavior is documented in https://everything.curl.dev/usingcurl/proxies/http.html#https-with-http-proxy.

[joyeecheung]

I'm not sure how impactful this is, as we shouldn't really encourage anyone to pass through a proxy with plain HTTP.

Agreed though I think from user's perspective, how the proxy or the endpoint works isn't usually in their control, so this might come up at some point for people who have to deal with this set up..the adoption of HTTPS vary greatly depending on which part of the world you are in (e.g. see https://www.ndss-symposium.org/wp-content/uploads/madweb25-1.pdf).

Anyway, I don't think it's urgent, just that it would be safer if we are aligned with curl eventually.

[caitp]

Curl provides the --proxytunnel option to allow opting into the always-tunnel behaviour even for unencrypted communication via an http proxy -- however I believe the tunnelling is required if using an https destination via an HTTP proxy, if I'm not mistaken -- which would be kind of the same thing here.

You could imagine situations where htp->http not sending a CONNECT could break some naive state management in some (proxy) servers, though, so maybe instead of opting into the tunneling for http via http proxy, we'd want to allow explicitly opting out it with an option. Just an idea