Project: Support Cronet Go binding

[klzgrad]

Any Go developers who are interested in using the core of naiveproxy in Go natively, please see my proposal and comment below:

The core of naiveproxy is namely Chromium's net stack. It's possible to expose its interface in the style of BSD socket in a library libnaive.so or libnaive.a (or more accurately libchromiumsocket.a), which then could be hooked up with a Go program via FFI. The benefit of doing this: no socks5 round trip overhead for using Chromium's net stack, and more versatile to link directly with Go's net stacks at socket/connection level. The impact of doing this: a better TLS fingerprint parrot than utls with similar Go development friendliness.

To achieve this goal:

• I can provide the code and build system to produce the proposed libnaive.a and corresponding C headers.
• An interested Go developer to discuss the interface definition for the Go FFI.
• Said Go developer develops Go binding and example integration showing Chromium net stack working in Go natively.

• Cronet's official Android build is for Java, but here the build is native and intended for use from Go. Do you @nekohasekai think it's better to use Cronet from Java or Go? I'll take it as a yes.
• The current build produces a shared library. Do you @nekohasekai think it's better to use a static library to link with Go?
• My Cronet build on Android still doesn't work.
• My Cronet build doesn't work with ThinLTO.
• I'm considering finalizing it to -connect-authority-override in the next version. Really hard to name this one. Currently it's decided as -connect-authority, as connect sufficiently defines the scope of this header, the leading - denotes its internal nature, and the suffix -override doesn't seem absolutely necessary and the lack of it doesn't introduce confusion.
• Record optimal static linking ldflags for each arch
• Add CI test case for cornet example and go example
• Document diff from upstream
• Write a specification of protocol used in Naiveproxy
• Add -network-isolation-key header to smuggle it in and configure stream isolation.

[nekohasekai]

Chromium already has cronet for calling from platforms like android, but it's basically https.

I'm not familiar with C/C++, but if you're done, I'll try it.

[klzgrad]

Cronet only provides a request-response API. One example: https://developer.android.com/guide/topics/connectivity/cronet/reference/org/chromium/net/UrlRequest.Builder.html#setHttpMethod(java.lang.String) the CONNECT method is not supported. I don't think there's any other way to create bidirectional sockets. Another proof is Cronet does not support WebSocket.

Cronet does support bidirectional "sockets". Several years ago when I checked, it was being used from gRPC with odd support status (ios only, or whatever, https://source.chromium.org/chromium/chromium/src/+/main:components/grpc_support/README.md: Currently Cronet (//components/cronet/ios) is the only consumer of this API). But it does support it. grpc-java does so via org.chromium.net.ExperimentalBidirectionalStream, grpc does it via third_party/objective_c/Cronet/bidirectional_stream_c.h (note the "objective_c", so it was designed for ios, but the cronet API is in C, which means this path may work but it was never tried). The reason I did not find it is because ExperimentalBidirectionalStream is not documented in https://developer.android.com/guide/topics/connectivity/cronet but it is documented in https://source.chromium.org/chromium/chromium/src/+/main:components/cronet/android/api/src/org/chromium/net/ExperimentalBidirectionalStream.java.

If you use Cronet's ExperimentalBidirectionalStream:

• Its traffic behavior goes through the same TLS code path with NaiveProxy and regular URL requests, so, same TLS behaviors. But on higher levels, internally it goes through BidirectionalStreamSpdyImpl, which is somewhat different from NaiveProxy's SpdyProxyClientSocket and http stream paths of regular URL requests. I'm just admitting imperfect parrot here, because I can't audit them exhaustively now within the large Chromium net stack.
• ExperimentalBidirectionalStream will need some care to avoid 1-RTT latency from waiting for HTTP response.
• ExperimentalBidirectionalStream is not exactly a socket. Some slightly more work is needed to handle its HTTP CONNECT logic. Auth headers.
• This happens in Java or C. It probably doesn't make sense to run heavy I/O between Java and Go at the same time. Assuming I/O heavy work happens in Go, then the work is to link Cronet native API to Go. The C API looks like this: https://source.chromium.org/chromium/chromium/src/+/main:components/grpc_support/include/bidirectional_stream_c.h. I will build a C++ test case to try this out before going with using Cronet's bidirectional stream.

The more ambitious goal of providing a general Go connection API with better TLS parroting seems better achieved with building a Go FFI with Cronet native API (if it works and is without major restrictions from the original Chromium stack). This requires more work, some design review, and more testing.

The less ambitious goal is to provide a specific Go API for running the logic of NaiveProxy inside Go address space without socks5 round trips, as a proxy backend along side other backends. This requires less work and is easier.

Recap of NaiveProxy

The changes made to Chromium

• Build minimization
  c647d3d base: Remove JNI function on Android
  b3a4768 dns: Fix iwyu
  0ac6ca7 libc++: Disable exceptions and RTTI
  543d0bd url: Remove perfetto tracing
  ecd4991 base: Disable trace event
  093898a base: Don't fix Y2038 problem with icu
  be723d4 net, url: Remove icu
  1c639b0 build: Disable Android java templates
  1c30676 build: Disable build_with_chromium
  27de40a base: Add Android stubs
  6750e63 net: Add Android stubs
  1fa442a build: Remove tests and minimize
  79fecf9 cert: Use builtin verifier on Android and Linux
  8708bc1 cert: Add SystemTrustStoreStaticUnix

• OpenWrt build support
  e3ef20e Fix OpenWrt build with use_allocator_shim=false
  1894111 allocator: Improve MIPS coverage of spinlocks
  61512a2 debug: Fix uClibc macro condition
  7e3a06f third_party: Fix missing sgidefs.h for Musl
  7104046 base: Do not forward declare stat64 for Musl
  7105b1e base: Fix narrowing casting for Musl
  eea4cbe base: Disable __close overloading for Musl
  5cfc396 process: Remove use of mallinfo for Musl
  d43ad9e base: Remove use of mallinfo for Musl
  489ba5d udp: Fix mmsghdr struct initializer for Musl
  2c0f6ff dns: Support Musl
  07dedde build: Add OpenWrt toolchains
  de5f764 build: Support MIPS -mtune= flag
  9df671f build: Support ARM build without FPU
  2c44fd6 build: Support -mcpu= on ARM and ARM64
  a8a02e3 libc++: Guard C++20 atomic type aliases
  e543ca2 lss: Avoid naming conflict in fstatat64

• Android build support
  c6606de Fix android build

• CI/CD
  9297696 Add continuous integration and tests
  0d649f1 Add build scripts
  a8c41f6 Add source import tool
  a38b058 build: Handle empty pkgconfig in sysroot
  b62139c build: Add sysroot creator script
  03f0fcf build: Force determinism in official build
  8c4e850 Add .gitignore

• Docs
  d07b4a5 Add example config.json
  a754cb6 Add README
  bc51961 Add LICENSE

• Naive itself: bf42527 Add initial implementation of Naive client, a hopefully complete specification of what it does:

  • http_proxy_socket.cc: an HTTP proxy frontend that accepts HTTP requests from clients (features: CONNECT method only; detects client opt-in of payload padding using padding header in the HTTP CONNECT request; advertise padding capability using padding header in the HTTP CONNECT response)
  • socks5_server_socket.cc: a SOCKS5 proxy frontend that accepts SOCKS5 requests from clients
  • naive_connection.cc: a bidirectional pipe that connects a client socket accepted from a frontend (socks5, http, redir) with Chromium's ClientSocketHandle which is initiated towards the requested proxy origin (features: payload padding encapsulation and decapsulation; )
  • naive_proxy.cc: Configures Chromium proxy settings, TLS params, parallelism with network isolation keys; A socket server that accepts TCP connections and moves them to backend
  • naive_proxy_bin.cc: Configures Chromium ClientSocketPool limits, config parsing, configures logging, builds URLRequestContext
  • naive_proxy_delegate.cc: Detects downstream and upstream padding capability and enables Fast Open logic in H2/H3 code in Chromium net stack.
  • redirect_resolver.cc: Deprecated, don't care

• Changes of Chromium required by Naive
  910fcc5 debug: Fix obsolete max check
  0198ac5 quic: Add support for HTTP/3 CONNECT Fast Open
  5f91e9d h2: Pad RST_STREAM frames
  a9bdea4 h2: Add support for HTTP/2 CONNECT Fast Open
  b95497e h2: Reduce warnings about RST on invalid streams
  6fe5dc1 socket: Force tunneling for all sockets
  3c48a78 socket: Allow higher limits for proxies
  f8e486b socket: Add RawConnect method
  bcd5d09 cert: Handle AIA response in PKCS#7 format

• Chromium itself
  1c761bb Import chromium-100.0.4896.60

Apparently much of Naiveproxy itself can be implemented with less and cleaner Go code.

Key changes and uses of the Chromium net stack, summary

• Adds support in Chromium's H2 and H3 code to support HTTP CONNECT "Fast Open", i.e. does not wait for the response after sending the HTTP CONNECT request and sends the payload immediately.
• Pads H2's RST_STREAM frames to slightly increase packet length entropy.
• "Handles AIA response in PKCS#7 format", otherwise it complains about issues dealing with Let's Encrypt's certificates, but it hasn't been thoroughly investigated whether this is necessary.
• Exposes a "RawConnect" function that creates Chromium ClientSocketHandle directly.
• Configures Chromium's proxy setting, QUIC stack versions, logging, extra headers, DNS resolver static mapping,

[klzgrad]

Cronet's BidirectionalStream:

• With proxy setting via myproxy.com, a BidirectionalStream to https://example.com will send TLS and HTTP headers to example.com, before sending user agent data to example.com. This does not work with the model where the user agent (browser) requires a socket to send TLS and HTTP headers to example.com for E2E encryption. (Cronet is the intermediary, not user agent)
• Without proxy settings, a BidirectionalStream to myproxy.com cannot use the CONNECT method to create HTTP tunnels, because the authority field can only be myproxy.com. Patching it to support this use case is very messy and not realistic, because the entire concept of "stream" in Chromium does not consider tunnels.

So the use case BidirectionalStream can support is something like WebSockets, where the origin is represented in other fields.

In NaiveProxy both the client intermediary and server intermediary assume HTTP/2 CONNECT tunneling. A proxy system using Cronet's BidirectionalStream won't work with Caddy's forwardproxy (without abusing HTTP headers).

TBC

[klzgrad]

Patching it to support this use case is very messy and not realistic
Cronet's BidirectionalStream won't work with Caddy's forwardproxy (without abusing HTTP headers)

I managed to find a not very messy way to smuggle the authority in: Pass a _real_authority HTTP header and use its value to override the authority when constructing the CONNECT header, and delete this fake header so it is not transmitted. This way a CONNECT tunnel can be set up, and then there is Cronet bidirectional stream API for simple read and write. With this, Cronet can be used as a substitute the "core" of Naiveproxy without the "major restrictions" mentioned above.

Architecturally Cronet and Naiveproxy both use the Chromium net core, but Cronet uses BidirectionalStream and Naiveproxy uses SpdyProxyClientSocket/QuicProxyClientSocket. BidirectionalStream and SpdyProxyClientSocket are also very similar. Naiveproxy is a more "lightweight" use of the Chromium net core because it can make more assumptions and use more simplifications. Cronet exposes more complete control interfaces.

In terms of the API as a product, Cronet is a better option because its API is better maintained. So unless more major restrictions are discovered (e.g. parrot detectors), it's better to use Cronet as basis to build proxy apps.

With that said, it's still better to use my fork of Chromium to build Cronet, because the build is minimized and supports OpenWrt and other niceties.

	Official Cronet release	My Cronet fork
Supported platforms	Android, iOS	Linux, Openwrt, Windows, Android, MacOS
Supported API	Java, ObjC	C
Usage	No CONNECT tunnels	Supports CONNECT tunnels

@nekohasekai The following is an example of using Cronet bidirectional stream:

naiveproxy-cronet-test-linux-x64.tar.xz from https://github.com/klzgrad/naiveproxy/releases/tag/cronet-test

# Note this does not use Chromium's build system
g++ bidi_example.cc libcronet.100.0.4896.60.so
# SSLKEYLOGFILE is saved at /tmp/keys so Wireshark can use it.
LD_LIBRARY_PATH=$PWD ./a.out https://my-caddy-forwardproxy.com "Basic $(printf user:pass | base64)"

Remaining questions:

[nekohasekai]

I have just tried

https://gist.github.com/nekohasekai/dda9ba499332433ba7a94f5f3d690a53

$ LD_LIBRARY_PATH=$PWD go run .
[0503/130502.548494:ERROR:cert_verify_proc_builtin.cc(603)] No net_fetcher for performing AIA chasing.
2022/05/03 13:05:02 on_stream_ready_callback
2022/05/03 13:05:02 on_response_headers_received, negotiated_protocol= h2
2022/05/03 13:05:02 :status:302
2022/05/03 13:05:02 cache-control:no-cache
2022/05/03 13:05:02 pragma:no-cache
2022/05/03 13:05:02 content-length:149
2022/05/03 13:05:02 content-type:text/html; charset=utf-8
2022/05/03 13:05:02 expires:-1
2022/05/03 13:05:02 location:https://www4.bing.com/?form=DCDN
2022/05/03 13:05:02 p3p:CP="NON UNI COM NAV STA LOC CURa DEVa PSAa PSDa OUR IND"
2022/05/03 13:05:02 set-cookie:SUID=M; domain=.bing.com; expires=Wed, 04-May-2022 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:MUID=2B8BC37FAEB966EC2616D2E6AFB167E3; domain=.bing.com; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; secure; SameSite=None
2022/05/03 13:05:02 set-cookie:MUIDB=2B8BC37FAEB966EC2616D2E6AFB167E3; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:_EDGE_S=F=1&SID=241DA119413A6A0C1EBCB08040326B77; domain=.bing.com; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:_EDGE_V=1; domain=.bing.com; expires=Sun, 28-May-2023 05:05:02 GMT; path=/; HttpOnly
2022/05/03 13:05:02 set-cookie:SRCHD=AF=NOFORM; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHUID=V=2&GUID=FEA003B5EF0546E1B9F9EC8783811781&dmnchg=1; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHUSR=DOB=20220503; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:SRCHHPGUSR=SRCHLANG=zh-Hant; domain=.bing.com; expires=Fri, 03-May-2024 05:05:02 GMT; path=/
2022/05/03 13:05:02 set-cookie:_SS=SID=241DA119413A6A0C1EBCB08040326B77; domain=.bing.com; path=/
2022/05/03 13:05:02 x-snr-routing:1
2022/05/03 13:05:02 strict-transport-security:max-age=31536000; includeSubDomains; preload
2022/05/03 13:05:02 x-cache:CONFIG_NOCACHE
2022/05/03 13:05:02 accept-ch:Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Mobile, Sec-CH-UA-Model, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version
2022/05/03 13:05:02 x-msedge-ref:Ref A: 03C48A42AE58420984C42B2E9D1FB30E Ref B: HKBEDGE0513 Ref C: 2022-05-03T05:05:02Z
2022/05/03 13:05:02 date:Tue, 03 May 2022 05:05:01 GMT
2022/05/03 13:05:02 on_read_completed
2022/05/03 13:05:02 <html><head><title>Object moved</title></head><body>
<h2>Object moved to <a href="https://www4.bing.com/?form=DCDN">here</a>.</h2>
</body></html>

2022/05/03 13:06:05 on_failed
2022/05/03 13:06:05 net error  -365

Seems to work, but looks like it requires a lot of wrappers to use in go

[nekohasekai]

When I use bidirectional_stream_write, it just returns 1 and then segmentation fault.

Fixed

[klzgrad]

looks like it requires a lot of wrappers to use in go

I don't know how many is a lot. 100-200 lines of code seem reasonable.

There can be something to do on the C side to make things nicer. I can think of this golang/go@918396b _GoString_ which helps a bit. What are the specific complexities in wrapping it? I can check if there are other changes in C that makes things easier in Go.

[nekohasekai]

I don't know how many is a lot. 100-200 lines of code seem reasonable.

I am writing some wrapper like this:

[nekohasekai]

Unless we have a code generator for that idl, I guess

Maybe https://chromium.googlesource.com/chromium/src.git/+/HEAD/mojo/README.md

[klzgrad]

Chromium's Mojo does not support Go bindings.

I kind of get what you're trying to do. cronet.idl exports 100~200 functions. An automatic binding generator wouldn't produce code as nice as that of manually written, which is not a small undertaking but also achievable with a few days of work. And once written, it's also a complete and general Go binding on its own, for the official Cronet library.

I would suggest starting with the important part of the API first so to prove it works, in terms of replacing libnaive.so with native use of Cronet. Even though it can be fun to engage in completionism, as I can imagine, PublicKeyPins is not needed for proving the above concept.

[nekohasekai]

Does the port in real-authority effect? I always get "Client sent an HTTP request to an HTTPS server."

[klzgrad]

https://github.com/klzgrad/naiveproxy/blob/cronet-test/src/net/spdy/spdy_http_utils.cc#L110

I changed it to _real_authority some time ago.

[nekohasekai]

Now requests to 443 works, but requests with _real_authority port 80 always fails directly with -9 error. what's wrong?

And if I call write before on_response_headers_received I will get the same error

[klzgrad]

Net errors are defined here https://source.chromium.org/chromium/chromium/src/+/main:net/base/net_error_list.h. But ERR_UNEXPECTED is not informative.

Try Wireshark with /tmp/keys to see what happens on the wire.